Introduction to Data Engineering on Google Cloud

In this course, you learn about data engineering on Google Cloud, the roles and responsibilities of data engineers, and how those map to offerings provided by Google Cloud. You also learn about ways to address data engineering challenges.

Syllabus

Course Introduction

Course Introduction

Data Engineering Tasks and Components

Module Introduction
The Role of a Data Engineer
Data Sources Versus Data Sinks
Data Formats
Storage Solution Options on Google Cloud
Metadata Management Options on Google Cloud
Sharing Datasets using Analytics Hub
Lab Intro: Loading Data into BigQuery
Loading data into BigQuery
Quiz

Data Replication and Migration

Module Introduction
Replication and Migration Architecture
The gcloud Command Line Tool
Moving Datasets
Datastream
Lab Intro: Datastream: PostgreSQL Replication to BigQuery
Datastream: PostgreSQL Replication to BigQuery
Quiz

The Extract and Load Data Pipeline Pattern

Module Introduction
Extract and Load Architecture
The bq Command Line Tool
BigQuery Data Transfer Service
BigLake
Lab Intro: BigLake: Qwik Start
BigLake: Qwik Start
Quiz

The Extract, Load, and Transform Data Pipeline Pattern

Module Introduction
Extract, Load, and Transform (ELT) Architecture
SQL Scripting and Scheduling with BigQuery
Dataform
Lab Intro: Create and Execute a SQL Workflow in Dataform
Create and execute a SQL workflow in Dataform
Quiz

The Extract, Transform, and Load Data Pipeline Pattern

Module Introduction
Extract, Transform, and Load (ETL) Architecture
Google Cloud GUI Tools for ETL Data Pipelines
Batch Data Processing Using Dataproc
Lab Intro: Use Dataproc Serverless for Spark to Load BigQuery
Use Serverless for Apache Spark to Load BigQuery
Streaming Data Processing Options
Bigtable and Data Pipelines
Lab Intro: Creating a Streaming Data Pipeline for a Real-Time Dashboard with Dataflow
Creating a Streaming Data Pipeline for a Real-Time Dashboard with Dataflow
Quiz