Lakeflow Connect - Smarter, Simpler File Ingestion With the Next Generation of Auto Loader

Explore the next generation of Auto Loader, Databricks' definitive tool for ingesting data from cloud storage into your lakehouse. Discover new features and best practices that simplify every aspect of cloud storage ingestion, including out-of-the-box observability for pipeline health and data quality monitoring. Learn about improvements in schema management, support for new data formats, and recent performance enhancements in Auto Loader. Gain insights into optimizing cost and performance through practical examples and proven best practices. Get a preview of upcoming features including a REST API for pushing files directly to Delta and a user interface for creating cloud storage pipelines. Presented by Chavdar Botev, Sr Staff Software Engineer, and Sandip Agarwala, Staff Software Engineer from Databricks, this session provides comprehensive coverage of file ingestion capabilities and future developments on the Databricks platform.