Coursera Flash Sale
40% Off Coursera Plus for 3 Months!
Grab it
Learn how to build a comprehensive real-time analytics pipeline for processing high-frequency IoT device data in this 37-minute conference talk from Databricks. Discover the implementation of a robust API endpoint that interfaces directly with IoT devices to process messages in real time from millions of distributed smart meters and IoT devices across networks. Explore the architecture that leverages cloud storage as a landing zone for raw data, followed by a streaming pipeline built on Delta Live Tables (DLT) implementing a multi-layer medallion architecture to progressively clean, transform, and enrich data. Understand how the pipeline operates continuously to maintain near real-time data freshness in gold layer tables that connect directly to Databricks Dashboards, providing stakeholders with immediate insights into operational metrics. Gain insights into modern data architecture approaches for handling high-volume IoT data streams while maintaining data quality and delivering accessible real-time analytics for business users. The session is presented by Nayan Sharma, Lead Data Engineer at CKDelta, and Padraic Kirrane, Data Scientist at CK Delta, who share practical implementation strategies and architectural decisions for production-ready IoT data processing solutions.
