Training AI Models on Massive Datasets

Learn how to train machine learning models on massive datasets that exceed memory limitations in this 31-minute conference talk from DevConf.US 2025. Discover practical strategies for handling large-scale data when datasets don't fit in main memory and GPU memory constraints pose additional challenges. Explore DASF, a comprehensive framework that combines lazy data loading techniques using Dask with acceleration methods through RAPIDS AI, enabling efficient processing of large datasets in ML pipelines across local and HPC environments. Gain insights from a real-world showcase demonstrating these techniques in action with an oil and gas sector company, providing concrete examples of how to overcome memory limitations while maintaining performance in machine learning workflows.