Building Scalable and Effective Data Pipelines with Argo for Large-Scale Metadata Processing

Watch a 24-minute conference talk exploring how Atlan's data team revolutionized their ETL pipeline performance using Argo workflows. Learn how they overcame challenges of massive data processing by implementing parallelization techniques, retry mechanisms, and effective artifactory loading. Discover the technical journey that led to processing 100 million assets with a 450% reduction in processing time and handling 3 million SQL queries in just 2 hours. Gain insights into scaling data pipelines effectively as speakers detail how they achieved a 50% reduction in overall pipeline runtime through Argo-powered horizontal scaling solutions.