Saving Millions From Millions - Navigating Towards Cost-Efficiency in Pinterest's Spark Jobs

Learn how Pinterest achieved tens of millions of dollars in cost savings while running millions of Apache Spark jobs monthly in this conference talk from Databricks. Discover analytical methodologies for identifying performance bottlenecks in large-scale Spark operations and explore technical solutions that address cost-efficiency challenges at massive scale. Examine strategies for extracting insights from billions of collected metrics and understand how remote shuffle services can address shuffle slowness, improve memory utilization, and reduce costs while managing hundreds of millions of pods. Gain practical knowledge about maintaining infrastructure cost efficiency to support rapid business growth and explore approaches that can help tackle common cost optimization challenges in the Apache Spark community.