Cross-Platform Data Lineage with OpenLineage - A Foundational Layer for Data Reliability

Learn how to implement cross-platform data lineage using OpenLineage as a foundational layer for data reliability in this 30-minute conference talk from SREcon25 Europe/Middle East/Africa. Discover why end-to-end lineage is essential for observability, debugging, and building trust in Data & AI workflows, and understand how software engineers can maintain the health of data pipelines using principles familiar to the SRE community. Explore the OpenLineage standard, a project under LF AI & Data, and see how it compares and complements OpenTelemetry while examining current integrations with popular tools including Airflow, Spark, dbt, and Flink. Gain insights from a Technical Steering Committee member of OpenLineage who presents both practical implementation strategies and advocates for treating Data & AI pipelines as first-class citizens in reliable, scalable systems, emphasizing the need to apply the same engineering rigor to data pipelines as production services.