Data Storage and Management for Big Data

Microsoft via Coursera

Go to class Write review

Overview

Coursera Flash Sale

40% Off Coursera Plus for 3 Months!

Grab it

This course provides a comprehensive overview of data storage and management approaches for big data. Learners will explore structured, semi-structured, and unstructured data formats, compare SQL and NoSQL database technologies, and implement data lakes and data warehouses. The course includes working with various file formats and understanding the differences between batch and real-time processing approaches. Course Learning Objectives: By the end of this course, you will be able to: - Compare and implement SQL and NoSQL database solutions for different big data scenarios - Work effectively with structured, semi-structured, and unstructured data formats - Design and implement data lakes and data warehouses for big data workloads - Build data pipelines using ETL and ELT approaches with Azure Data Factory - Differentiate between batch and real-time processing methodologies and implement appropriate solutions

Syllabus

Data Storage Technologies (SQL vs NoSQL)

Data Storage Technologies (SQL vs NoSQL) guides learners through the core principles of modern data storage and the trade-offs that shape today’s big data systems. The module examines how relational databases manage structured data, where they encounter limitations at scale, and how techniques such as partitioning, indexing, and lakehouse architectures mitigate performance gaps. Learners compare major NoSQL categories—including document, key-value, and column-family databases—to understand how flexible schemas and distributed designs support high-volume, high-velocity workloads. Through hands-on activities with SQL Server, Azure Synapse, and Azure Cosmos DB, learners practice essential operations, evaluate storage technologies based on workload requirements, and build the skills needed to select and implement effective database solutions for big data environments.

Working with Data Formats (Structured, Semi-structured, Unstructured)

Working with Data Formats (Structured, Semi-structured, Unstructured) helps learners build a clear understanding of how different data formats function within big data systems and why format selection matters for performance, storage, and analytical success. The module introduces structured formats, such as CSV and TSV, and explores flexible semi-structured formats, including JSON and XML. It also examines optimized file types, including Parquet, Avro, and ORC, that support large-scale analytics. Learners practice transforming data between formats using Azure Data Factory, working with nested structures, applying schema inference, and evaluating performance trade-offs across file types. Through demonstrations, code exercises, and hands-on labs, this module equips learners to select, convert, and manage data formats effectively for diverse big data scenarios.

Data Lakes and Data Warehouses Implementation

Data Lakes and Data Warehouses Implementation guides learners through the architectural foundations and hands-on skills needed to build modern analytical environments. The module explores the purpose and structure of data lakes, highlighting the zones of raw, cleaned, enriched, and curated data, and demonstrates how thoughtful design supports flexibility, governance, and large-scale analytics. Learners also study core data warehouse concepts, including dimensional modeling, star schemas, and data marts, to understand how structured storage enables high-performance querying. Through practical work with Azure Data Lake Storage Gen2 and Azure Synapse Analytics, learners design zone architectures, implement dimensional models, configure SQL pools, and apply best practices for partitioning, distribution, and optimization. By the end, they gain the ability to organize, govern, and integrate data across both lake and warehouse environments, supporting scalable, enterprise-ready analytics.

Building Data Pipelines (ETL/ELT with Azure Data Factory)

Building Data Pipelines (ETL/ELT with Azure Data Factory) equips learners with the skills to design, implement, and manage scalable data integration workflows using modern, cloud-native approaches. The module examines the differences between ETL and ELT, helping learners understand when each methodology delivers the best performance, flexibility, and cost efficiency. Learners gain hands-on experience with Azure Data Factory, configuring linked services, datasets, activities, and core orchestration components, and practice building both simple and advanced pipelines. The module also introduces transformation logic, control flow patterns, parameterization, and error handling strategies that support production-ready data engineering solutions. Through walkthroughs, labs, code exercises, and scenario-based decisions, learners learn to monitor pipelines, troubleshoot failures, and design reliable data workflows that support enterprise-scale analytics.

Batch and Real-Time Processing Fundamentals

Batch and Real-Time Processing Fundamentals introduces learners to the core processing models that power modern big data systems, helping them understand when each approach delivers the most value. The module explores batch architectures, scheduling methods, and optimization strategies for large-scale historical processing, while also examining real-time stream processing concepts, including event handling, latency trade-offs, and throughput requirements. Learners gain hands-on experience implementing both models—building batch workflows with Azure Data Factory and configuring streaming pipelines using Event Hubs and Stream Analytics. Through architectural analysis, code exercises, and practical labs, learners learn to evaluate business needs, select the right processing approach, and design hybrid systems that combine batch and streaming for comprehensive analytics.