Realtime Voting System - End-to-End Data Engineering Project

Embark on a comprehensive end-to-end data engineering project to build a real-time voting system using modern big data technologies such as Apache Kafka, Apache Spark, and Streamlit. Learn to design and implement the election flow, set up the system architecture, and develop key components including candidate and voter accreditation, vote streaming, data preprocessing, and aggregation. Create a real-time dashboard to visualize election results and live commentaries. Gain hands-on experience with Docker, PySpark, and Streamlit while exploring the intricacies of building a robust, scalable voting system for modern elections.

Syllabus

Introduction
The Election Flow
The System Architecture
Setting up the Project
The docker compose for the system architecture
Candidate, Party and Voters Accredditation
Voting System
Streaming votes from/to Kafka
Reading Voting Data Streams from Kafka with Pyspark
Spark Data preprocessing
Spark Aggregations
Writing aggregated data back to kafka
Realtime Votes Dashboard with Streamlit
Results and Live commentaries
Winner of the Election
31 Outro