Big Data Foundations with Hadoop and Spark

Packt via Coursera Specialization

Go to class Write review

Details

Go to class

Provider

Coursera Specialization
Pricing

Paid Course
Languages

English
Certificate

Certificate Available
Duration & workload

4 weeks, 10 hours a week

Found in

Overview

Coursera Flash Sale

40% Off Coursera Plus for 3 Months!

Grab it

This specialization features Coursera Coach! A smarter way to learn with interactive, real-time conversations that help you test your knowledge, challenge assumptions, and deepen your understanding as you progress through the specialization. Big data is transforming industries, and this specialization equips you with the skills to succeed. You’ll gain a foundation in Hadoop and Spark, learning how to store, process, and analyze massive datasets. Through theory and hands-on projects, you’ll develop practical expertise that applies directly to real-world scenarios. You’ll begin with Hadoop, setting up the Hortonworks Sandbox and working with HDFS, MapReduce, Pig, Hive, and Spark. Then, you’ll move to Apache Spark with Scala, mastering RDDs, SparkSQL, DataFrames, and cluster optimization. Next, you’ll explore Spark Streaming to process live data with use cases like Twitter analysis and log tracking. The specialization concludes with Apache Kafka, where you’ll implement producers, consumers, and advanced operations such as KRaft mode and Kafka Java programming. This intermediate specialization is designed for learners with basic programming in Java, Python, or Scala. It is ideal for data engineers, developers, and aspiring data scientists. By the end of the specialization, you will be able to design big data pipelines, process batch and real-time data, and integrate Kafka for scalable applications.

Syllabus

Course 1: The Ultimate Hands-On Hadoop
Course 2: Apache Spark with Scala – Hands-On with Big Data!
Course 3: Streaming Big Data with Spark Streaming, Scala, and Spark 3!
Course 4: Apache Kafka Series - Learn Apache Kafka for Beginners v3

Courses

0 reviews
14 hours 51 minutes
View details

Updated in May 2025. This course now features Coursera Coach! A smarter way to learn with interactive, real-time conversations that help you test your knowledge, challenge assumptions, and deepen your understanding as you progress through the course. In this course, you'll embark on a journey to master Apache Kafka, beginning with a solid introduction to its core concepts and architecture. You'll learn about essential topics such as producers, consumers, partitions, and offsets, all while understanding how they fit together within the broader Kafka ecosystem. Each theoretical section is paired with practical examples, ensuring that you not only understand the "what" but also the "how" of Kafka. As you progress, you'll delve into advanced Kafka operations, including starting Kafka without Zookeeper using KRaft mode and mastering the command line interface. The course also includes a thorough exploration of Kafka Java programming, where you'll learn to implement producers and consumers, handle callback functions, and manage consumer groups. By the end of this section, you'll be ready to tackle real-world Kafka projects, armed with the knowledge to implement efficient, scalable solutions. The final modules focus on real-world insights and case studies, where you'll see how Kafka is applied in different industries, from big data ingestion to logging and metrics aggregation. You'll also explore Kafka's role in enterprise environments, focusing on cluster setup, security, and multi-cluster operations. Whether you're new to Kafka or looking to deepen your understanding, this course provides the tools and knowledge to excel in the world of data streaming. This course is ideal for software developers, data engineers, and IT professionals who are new to Apache Kafka and wish to learn from the ground up. No prior experience with Kafka is required, though a basic understanding of distributed systems and Java programming is beneficial.
0 reviews
11 hours 11 minutes
View details

Embark on a journey to master big data processing with Apache Spark and Scala. This course begins with setting up your development environment, ensuring you have a solid foundation in both Spark and Scala. You will dive into a Scala crash course that covers syntax, flow control, functions, and data structures, giving you the essential skills needed to work with Spark. Next, you will explore Spark's core concept, the Resilient Distributed Dataset (RDD). Through a series of hands-on activities and exercises, you will learn to manipulate RDDs, implement key/value operations, and perform complex data transformations. The course then transitions into SparkSQL, DataFrames, and DataSets, where you will practice querying structured data efficiently. You'll also tackle advanced Spark programming, where you’ll apply algorithms to real-world datasets, work with clusters, and optimize performance. As you progress, you will delve into machine learning with Spark MLlib and explore how to build recommendation systems, perform regression analysis, and implement decision trees. Finally, the course introduces Spark Streaming and GraphX, allowing you to process real-time data streams and graph-based data efficiently. By the end of this course, you will have the expertise to leverage Spark and Scala for complex data processing tasks in any industry. This course is designed for software engineers who want to expand their skills into the world of big data processing on a cluster. It is necessary to have some prior programming or scripting knowledge.
0 reviews
8 hours 51 minutes
View details

Updated in May 2025. This course now features Coursera Coach! A smarter way to learn with interactive, real-time conversations that help you test your knowledge, challenge assumptions, and deepen your understanding as you progress through the course. In the fast-evolving world of big data, the ability to process streaming data in real time is essential. This course is meticulously designed to take you from the basics of Spark and Scala to advanced real-time data processing with Spark Streaming. We begin with a foundational setup of your development environment, ensuring you are equipped to run Spark and Scala on your desktop. A hands-on activity will introduce you to the excitement of live data by streaming and analyzing real-time Tweets. As we move forward, you’ll gain a solid understanding of Scala, a language integral to working with Spark. This crash course in Scala covers the essentials: variables, data structures, and flow control, with practical exercises to cement your understanding. With a firm grip on Scala, you’ll delve into the core concepts of Spark, including the Resilient Distributed Dataset (RDD), which forms the backbone of Spark Streaming applications. We will then explore Spark Streaming in detail, from its architecture to fault tolerance mechanisms, using engaging examples like tracking Twitter hashtags and analyzing Apache logs. Finally, the course pushes the boundaries of your knowledge with advanced topics such as integrating Spark Streaming with Kafka, Flume, and Cassandra. You'll also tackle stateful information tracking, real-time machine learning with K-Means clustering, and deploying your applications on a real Hadoop cluster. By the end of this course, you’ll not only understand the theory behind Spark Streaming but will have the practical experience to apply it effectively in production environments. This course is ideal for software developers, data engineers, and data scientists with a basic understanding of programming concepts. Prior experience with Java, Python, or any object-oriented programming language is recommended but not required. Familiarity with big data concepts will be helpful but is not mandatory.
0 reviews
19 hours 47 minutes
View details

Updated in May 2025. This course now features Coursera Coach — your interactive learning companion that helps you test your knowledge, challenge assumptions, and deepen your understanding as you progress. Build a strong, hands-on foundation in Hadoop and big data processing with this comprehensive course designed for data engineers, developers, and IT professionals. From installation to advanced analytics, you’ll learn how to work confidently with Hadoop’s ecosystem and design scalable solutions for real-world data challenges. You’ll begin by installing the Hortonworks Data Platform (HDP) Sandbox on your local machine, giving you an isolated environment to explore Hadoop’s core components. Through guided exercises, you’ll work with the Hadoop Distributed File System (HDFS) and build your understanding of MapReduce, learning how large-scale distributed processing works behind the scenes. As you progress, you’ll move into advanced Hadoop programming with Pig, Hive, and Spark. You’ll write complex queries, analyze large datasets, and work with real-world data to build scalable data workflows. You’ll also explore machine learning with Spark MLLib, giving you a practical introduction to distributed ML techniques. In the final modules, you’ll learn how to manage and optimize Hadoop clusters using YARN, ZooKeeper, Oozie, and Kafka. You’ll practice feeding data into your cluster, orchestrating workflows, managing resources, and analyzing streaming data in real time — essential skills for production-grade environments. By the end of this course, you will have: - Installed and configured the Hortonworks Sandbox for Hadoop development. - Worked with HDFS, MapReduce, and Hadoop’s core data processing concepts. - Written queries and pipelines using Pig, Hive, and Spark. - Performed distributed machine learning with Spark MLLib. - Integrated relational and non-relational data sources with Hadoop. - Managed clusters and streaming workflows with YARN, ZooKeeper, Oozie, and Kafka. - Gained the confidence to design and implement Hadoop-based data solutions. This course is ideal for data engineers, developers, and IT professionals with basic programming or data management experience. Familiarity with Java, SQL, or the Linux command line is helpful but not required.