Google, IBM & Meta Certificates — All 10,000+ Courses at 40% Off
One annual plan covers every course and certificate on Coursera. 40% off for a limited time.
Get Full Access
Explore a cloud-native data ingestion architecture that significantly reduces costs while improving reliability and efficiency in this 32-minute conference talk from Databricks. Learn how Scribd transformed their data ingestion strategy by leveraging open-source tools like kafka-delta-ingest, oxbow, and Airbyte to create an event-driven system that ingests data from AWS Aurora, SQS, Kinesis Data Firehose, and other sources into Delta Lake. Discover how to eliminate the need for traditional jobs while building a more robust data platform that integrates seamlessly with Databricks and Unity Catalog environments. Gain insights into implementing third-party tools within a unified data architecture that delivers high efficiency and availability. While the presentation focuses on AWS implementation, understand how these principles can be adapted for Azure, Google Cloud Platform, or on-premise environments to create a cost-effective, scalable data ingestion solution.
