Google, IBM & Meta Certificates — All 10,000+ Courses at 40% Off
One annual plan covers every course and certificate on Coursera. 40% off for a limited time.
Get Full Access
Learn how to streamline data integration workflows using Databricks' newly developed Python Data Source API for Apache Spark in this 39-minute conference talk. Discover how data engineering teams can overcome the challenges of building custom ingest and egress solutions for proprietary, industry-specific, or non-standard data sources and sinks. Explore the comprehensive research process that led Databricks to interview companies across various industries to understand their diverse data integration needs and pain points. Understand how this feedback directly influenced the development of the Python Data Source API as a solution to reduce the cumbersome and time-consuming nature of custom data integration work. Gain insights into practical applications and implementation strategies for leveraging this new API to simplify your data engineering processes. The presentation is delivered by Craig Lukasik, Senior Solutions Architect at Databricks, who provides expert guidance on modernizing data pipeline development and reducing the complexity of handling diverse data sources within Apache Spark environments.
