Coursera Flash Sale
40% Off Coursera Plus for 3 Months!
Grab it
- Learn how to design and implement data modeling strategies in Azure Databricks with Unity Catalog, including ingestion patterns, table formats, partitioning, slowly changing dimensions, and clustering strategies.
By the end of this module, you'll be able to:
- Design data ingestion logic and configure data source connections
- Select the appropriate data ingestion tool for your scenario
- Choose between Delta Lake, Apache Iceberg, and other table formats
- Design and implement effective data partitioning schemes
- Select and implement slowly changing dimension types
- Design and implement temporal tables for change tracking and auditing
- Choose appropriate data granularity for fact and dimension tables
- Design and implement clustering strategies for query optimization
- Evaluate when to use managed versus external tables
- Learn how to ingest data from diverse sources into Unity Catalog tables in Azure Databricks using managed connectors, notebooks, SQL commands, streaming, and declarative pipelines.
By the end of this module, you'll be able to:
- Configure Lakeflow Connect to ingest data from external sources using managed connectors
- Ingest batch and streaming data using notebooks with DataFrames and Structured Streaming
- Use SQL commands like COPY INTO and CREATE TABLE AS SELECT for file-based ingestion
- Process change data capture feeds with the AUTO CDC API
- Configure Spark Structured Streaming for real-time data ingestion from Kafka and Event Hubs
- Set up Auto Loader to automatically detect and process new files with schema evolution
- Orchestrate data ingestion workflows using Lakeflow Spark Declarative Pipelines
- Learn how to cleanse, transform, and load data into Unity Catalog tables in Azure Databricks by profiling data, handling duplicates and nulls, applying transformations, and using various loading strategies.
By the end of this module, you'll be able to:
- Profile data using SQL commands and data profiling features to assess data quality
- Choose appropriate column data types to optimize storage and ensure data integrity
- Identify and resolve duplicate, missing, and null values in datasets
- Apply filtering, grouping, and aggregation operations to transform data
- Combine datasets using joins and set operators like UNION, INTERSECT, and EXCEPT
- Reshape data using denormalization, pivot, and unpivot techniques
- Load transformed data into Unity Catalog tables using INSERT, MERGE, and overwrite operations
- Learn how to implement and manage data quality constraints in Azure Databricks using Unity Catalog, including validation checks, schema enforcement, and pipeline expectations.
By the end of this module, you'll be able to:
- Implement validation checks for nullability, cardinality, and range constraints
- Implement data type checks using schema enforcement and explicit casting
- Enforce schema and manage schema drift using Auto Loader and Delta Lake
- Manage data quality using pipeline expectations in Lakeflow Spark Declarative Pipelines
