The Fundamentals of Type-safe, Reproducible, and Scalable Data Pipelines

Learn how to create robust, reproducible, and scalable data science and machine learning pipelines in this tech talk presented by Niels Bantilan from Union.ai. Discover the fundamentals of building DS/ML pipelines using Flytekit through task composition and workflow creation, while exploring how Python's typing module enables type safety and automatic data lineage tracking. Gain insights into Flyte's use of Docker containers for ensuring reproducibility, and master techniques for optimizing pipeline performance through resource configuration and task caching in Python. Perfect for data scientists and ML practitioners looking to bridge the gap between laboratory development and production deployment of models and analyses.