Overview

Google, IBM & Meta Certificates – 40% Off

One plan covers every Professional Certificate on Coursera.

This course builds a strong foundation in modern data engineering using Databricks, Apache Spark, and Lakehouse architecture. You will learn how data engineers design, process, organise, and ingest data to support analytics, AI, and machine learning workflows. You will begin by exploring the role of data engineering in modern organisations, including how it differs from data science and why reliable data pipelines are essential for AI/ML success. You will then get introduced to Databricks, its workspace environment, notebooks, and core platform capabilities. Next, you will work with Apache Spark and PySpark to process data at scale. You will explore DataFrames, transformations, schemas, supported file formats, and practical workflows for reading, transforming, and writing data in formats such as CSV, JSON, and Parquet. You will also examine modern storage concepts, including data lakes, data warehouses, and Lakehouse architecture. Through Databricks, you will learn how to organise data using catalogs, schemas, and volumes, and understand how structured data management supports scalable engineering workflows. By the end of this course, you will be able to: - Explain core data engineering concepts and the role of Databricks. - Describe Lakehouse architecture and modern data storage patterns. - Process and transform data using Apache Spark and PySpark. - Organise project data using catalogs, schemas, and volumes. - Ingest data through batch, API-based, and real-time workflows. Designed for aspiring data engineers, data analysts, software developers, and students entering the data field, this course prepares you to build a strong technical foundation for advanced Databricks, Delta Lake, and AI/ML pipeline development.

Syllabus

Introduction to Data Engineering and Databricks

Build a strong foundation in data engineering by exploring its role in modern analytics and AI/ML ecosystems. This module introduces the data engineering lifecycle, key architectural concepts such as data lakes, warehouses, and lakehouses, and provides hands-on experience with the Databricks platform. Learners will gain the knowledge needed to navigate Databricks, create notebooks, and understand how modern data platforms support scalable data and AI workloads.

Data Processing with Apache Spark and PySpark

Learn how modern data platforms process large-scale datasets using Apache Spark and PySpark. This module covers Spark’s core architecture, execution concepts, and DataFrame-based transformations while providing hands-on experience reading, processing, and storing data in common formats. Learners will build foundational skills for scalable data processing and analytics within the Databricks environment.

Organizing and Ingesting Data in the Databricks

Develop the foundational skills required to organize and ingest data in modern Databricks environments. This module explores Unity Catalog structures, including catalogs, schemas, and volumes, while introducing key data ingestion concepts, source systems, and ingestion patterns. Through hands-on exercises, learners will work with batch, streaming, and API-based ingestion workflows to build reliable and scalable data pipelines.