Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

End-to-End Big Data Engineering Project with Azure - Part 1

Krish Naik via YouTube

Overview

Coursera Flash Sale
40% Off Coursera Plus for 3 Months!
Grab it
Learn to build an end-to-end big data engineering project in this comprehensive video tutorial that covers Azure cloud services and data engineering fundamentals. Master essential concepts including Azure Data Factory, Azure Databricks, and ADLS storage while working with real-world datasets like Olist. Starting with project architecture and Azure account setup, progress through data ingestion techniques using SQL databases and MongoDB, implement Medallion Architecture, and create automated workflows. Explore both batch and real-time data processing, parametrized ingestion, and establish seamless connections between Databricks and Azure Data Lake Storage. Gain hands-on experience with resource management, compute configurations, and practical implementation of data engineering principles in a cloud environment. Perfect for professionals with 2+ years of experience looking to enhance their big data engineering skills.

Syllabus

Project Introduction
Getting Started
Pre-requisite
Project Architecture
Azure Free Account
Azure Cloud
Olist Dataset
SQL DB & Data Ingestion
Role of Data Engineer
Resource & Resource Group
Azure Data Factory
ADLS storage account
Meddallion Architecture
Ingestion with ADF
Real Time Ingestion with ADF
Parametrized Ingestion ADF
Azure Databricks Creation
Databricks Overview
Databricks UI Overview
Compute and Notebook
MongoDB & Ingestion
Databricks Workflow
Datbricks to ADLS connection
Accessing data from Lake
Outro & Next Part

Taught by

Krish Naik

Reviews

Start your review of End-to-End Big Data Engineering Project with Azure - Part 1

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.