Google, IBM & Meta Certificates — All 10,000+ Courses at 40% Off
One annual plan covers every course and certificate on Coursera. 40% off for a limited time.
Get Full Access
Explore a 28-minute conference talk that delves into innovative solutions for managing large-scale AI training datasets. Learn about Mosaic StreamingDataset, designed to simplify multi-node, distributed training of large models, and its seamless integration with PyTorch. Discover the advantages of Lance columnar format over parquet for ML workloads, offering significantly improved random access performance crucial for various training operations. Gain insights into how combining StreamingDataset with Lance enables direct data streaming from object storage, resulting in enhanced performance and cost-effectiveness. Examine the inner workings of these technologies and understand how to construct a basic training pipeline that leverages their capabilities for more efficient and higher-quality AI model training.
