Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

CNCF [Cloud Native Computing Foundation]

Running Big Data Applications at Scale on K8s

CNCF [Cloud Native Computing Foundation] via YouTube

Overview

Google, IBM & Meta Certificates — All 10,000+ Courses at 40% Off
One annual plan covers every course and certificate on Coursera. 40% off for a limited time.
Get Full Access
Explore how Intuit leverages Spark on Kubernetes to process big data at scale in this conference talk. Learn about the advantages of running data processing workloads on Kubernetes, including cost reduction and increased production speed. Discover Intuit's journey in building a data processing platform, their experiences with the Spark operator, and how they addressed challenges like network bottlenecks. Gain insights into the benefits of containerization for data scientists and the future plans for Intuit's big data infrastructure. Understand the comparison between Kubernetes and Yarn, native workflows, and the impact on transaction categorization and personalization problems.

Syllabus

Introduction
Intuits Data Lake
Kubernetes vs Yarn
Native workflow
Spark operator
Transaction categorization
Personalization problem
Learnings
SpoK Part 3
Advantages
Future plans
Contact details
Questions
Cost Reduction
Cost Advantages
Effort
Spark
Network bottlenecks
Spark Operator on Kubernetes
What new things should data scientists learn
Container Journey
Slides
Spark Containers
Feature Processing
Spark Overlay
What do data scientists need to learn
Wrap up

Taught by

CNCF [Cloud Native Computing Foundation]

Reviews

Start your review of Running Big Data Applications at Scale on K8s

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.