Data Warehousing and Integration Part 2

Northeastern University via Coursera

Go to class Write review

Overview

AI, Data Science & Cloud Certificates from Google, IBM & Meta — 40% Off

One plan covers every Professional Certificate on Coursera. 40% off Coursera Plus Annual.

Unlock All Certificates

Covers various topics in Data Engineering in support of decision support systems, data analytics, data mining, machine learning, and artificial intelligence. Studies on-premises data warehouse architecture, dimensional modeling of data warehouses, Extract-Transform-Load (ETL) integration from source systems to data warehouse, On-line Analytical Processing (OLAP) systems, and the evolving world of data quality and data governance. Offers students an opportunity to design, develop and maintain cloud-based data pipelines. Both on-premises and cloud-based platforms will be used to illustrate and implement Data Engineering techniques using operational and analytical data warehouses.

Syllabus

ETL Design 1

In this module, you'll learn about ETL (Extract, Transform, Load) processes, an essential part of Data Warehousing and Data Integration solutions. ETL processes can be complex and costly, but effective design and modeling can significantly reduce development and maintenance costs. You'll be introduced to the basics of Business Process Modeling Notation (BPMN), which is crucial for modeling business processes. We’ll focus on the basics of BPMN, including key components such as flow objects, gateways, events, and artifacts, which are essential for modeling business processes. You will explore how BPMN can be customized to conceptual modeling of ETL tasks, with a particular focus on differentiating control tasks from data tasks. Control tasks manage the orchestration of ETL processes, while data tasks handle data manipulation, both of which are critical in conceptualizing ETL workflows. By the end of this module, you’ll gain a solid understanding of how to design ETL processes using BPMN, enabling greater flexibility and adaptability across various tools.

ETL Design 2

In this module you will dive into Talend Studio, a powerful Eclipse-based data integration platform that transforms complex ETL operations into intuitive visual workflows. By explorating Talend's drag-and-drop interface, you will learn to navigate the core components of the platform. You’ll master fundamental ETL operations by studying essential components like tMap for complex data transformations and joins, tJoin for straightforward data linking, and various input/output components for connecting to databases, files, and APIs. By the end of the module you will understand how Talend automatically generates executable Java code from visual designs, enabling you to create scalable, production-ready data integration solutions that can handle both batch processing and real-time data scenarios across diverse technological environments.

Data Engineering 1

In this module, we transition from on-premises Data Warehousing to Data Engineering. While Data Engineering has its roots in Data Warehousing, it encompasses much more. We'll explore the key enablers of this evolution, specifically cloud computing and DevOps. You will learn about the benefits of cloud development, including enhanced scalability, cost efficiency, and flexibility in data operations. We will also dive into how traditional IT infrastructure components—such as security, networking, and compute resources—are redefined in cloud environments using AWS. Additionally, you'll gain an understanding of DevOps in the cloud, focusing on the use of virtual machines and containers to streamline continuous integration and deployment. We will cover key DevOps practices like Infrastructure as Code (IaC), CI/CD pipelines, and automated testing, emphasizing their role in ensuring consistency, faster development cycles, and secure applications. You will then explore what data engineering entails and the skills required to become a data engineer. Next, we'll introduce the concept of the data engineering lifecycle and its different phases, focusing on the first two: Data Generation and Storage. Finally, we'll bring these concepts into practice by introducing key AWS services used in data engineering and showing how a simple pipeline is built on AWS, from raw storage through to the data lakehouse architecture.

Data Engineering 2

In this module, we will explore the next two phases of the data engineering lifecycle: Ingestion and Transformation. Data ingestion refers to the process of moving data from source systems into storage, making it available for processing and analysis. As you delve into the reading, you will examine key ingestion patterns, including batch versus streaming ingestion, synchronous versus asynchronous methods, and push, pull, and hybrid approaches. You’ll also explore essential engineering considerations such as scalability, reliability, and data quality management, along with the challenges posed by schema changes. The reading will introduce various technologies that enable data ingestion, such as JDBC/ODBC, Change Data Capture (CDC), APIs, and event-streaming platforms like Kafka. We then shift focus to the transformation phase of the lifecycle, exploring different types of transformations that integrate complex business logic into data pipelines. At the end of the module, we will focus on data architecture and implementing good architecture principles to build scalable and reliable data pipelines.

Pipeline Planning

In this module, we will explore data characteristics and how they drive infrastructure decisions. In today’s data-driven world, understanding the properties of your data is essential for designing robust data pipelines. We’ll go over key characteristics like volume, which refers to the size of datasets, and velocity, which concerns how frequently new data is generated. We’ll also take a look at variety, which focuses on data formats and sources, and veracity, which emphasizes data accuracy and trustworthiness. The ultimate goal is to uncover value from data through insightful analysis. As we delve into pipeline design, you'll learn how these characteristics influence key decisions, such as the choice of storage, processing, and analytics tools. We will also cover AWS Lambda and event-driven architecture, exploring how serverless compute connects the storage and processing layers of a cloud pipeline. We will look at how IAM execution roles, S3 event triggers, and CloudWatch Logs work together to make pipelines reactive and observable. We will then extend that to Glue Jobs and Amazon QuickSight, completing the picture from raw ingestion through to analysis and visualization. By the end of this module, you’ll have a comprehensive understanding of how to build effective data solutions to meet both technical and business needs.

Serving Data

Welcome to the final stage of the data engineering lifecycle: serving data. In this module, we will focus on how to effectively serve data for analytics, machine learning (ML), and reverse ETL to ensure that the data products you design are reliable, actionable, and trusted by stakeholders. Key topics include setting SLAs, identifying use cases, evolving data products with feedback, standardizing data definitions, and exploring delivery methods such as file exchanges, databases, and streaming systems. We’ll also cover the use of reverse ETL to improve business processes and discuss the importance of context for choosing the best visualization type and tools. We then delve into KPIs and metrics and how to classify them, including how to identify robust KPIs based on the business context. Finally, we will focus on creating intuitive dashboards by choosing the right analysis, visualizations, and metrics to showcase based on the business context and audience involved. By the end of this module, you will understand how to design and serve data solutions that drive meaningful action and are trusted by end users.