Improve Your App UX with One Model Processing Insights from Batch and Streams

Explore a comprehensive talk on improving app user experience through unified batch and stream processing using Google's Dataflow model. Dive into the evolution of big data processing, from MapReduce to Apache Beam, and learn how to effectively manage and visualize data streams. Discover practical insights on implementing time-based windows, handling delays, and utilizing triggers in Beam. Follow along with a real-time demonstration showcasing Dataflow's capabilities, including pipeline creation, refinement, and visualization techniques. Gain valuable knowledge on integrating batch and streaming data processing to enhance your application's performance and user experience.

Syllabus

Introduction
Steps to improve user experience
History of data processing at Google
What is MapReduce
The problem with MapReduce
From Java
The problem
Artificial splitting
Un unbounded data
Delays
How to deal with delays
MillVia
Timebased windows
Session windows
Event vs processing time
Stream vs Batch
Billing Pipeline
User Experience
Abuse Detection
Historical Systems
Apache Beam
Dataflow Example
Four Questions
MapReduce
When to omit results
Create a window
Wait for results
When to trigger
Triggers in Beam
Demo
refinements
how
what just happened
cancel pipeline
run on
update pipeline
QR code
Assign color
Running the pipeline
Patch pipeline
BigQuery
Color Smash
Hit Ratio
Aggregate
Back to the slides