Coursera Flash Sale
40% Off Coursera Plus for 3 Months!
Grab it
Explore a cloud-native data ingestion architecture that significantly reduces costs while improving reliability and efficiency in this 32-minute conference talk from Databricks. Learn how Scribd transformed their data ingestion strategy by leveraging open-source tools like kafka-delta-ingest, oxbow, and Airbyte to create an event-driven system that ingests data from AWS Aurora, SQS, Kinesis Data Firehose, and other sources into Delta Lake. Discover how to eliminate the need for traditional jobs while building a more robust data platform that integrates seamlessly with Databricks and Unity Catalog environments. Gain insights into implementing third-party tools within a unified data architecture that delivers high efficiency and availability. While the presentation focuses on AWS implementation, understand how these principles can be adapted for Azure, Google Cloud Platform, or on-premise environments to create a cost-effective, scalable data ingestion solution.
