Generative AI Model Data Pre-Training on Kubernetes - A Use Case Study

Explore how to leverage Kubernetes and Kubeflow Pipelines for large-scale generative AI model data preprocessing in this 22-minute conference talk from DevConf.CZ 2025. Learn about the challenges of preprocessing petabytes of data for Large Language Models and discover how Kubeflow Pipelines provide flexibility, repeatability, and scalability for LLM data processing workflows used daily at IBM Research to build enterprise-focused indemnified LLMs. Compare different data preparation toolkits built on Kubernetes, Rust, Slurm, and Spark to understand how to choose the right approach for your LLM experiments or enterprise use cases. Examine how the open source Data Prep Toolkit utilizes KFP and KubeRay for scalable pipeline orchestration, including processes like deduplication, content classification, and tokenization. Gain insights from real-world experience with KFP implementation, including challenges faced and lessons learned, while understanding its broader applicability for diverse LLM tasks such as data preprocessing, RAG retrieval, and model fine-tuning.