Coursera Flash Sale
40% Off Coursera Plus for 3 Months!
Grab it
Learn how to build a reliable, high-velocity machine learning model serving platform through this 31-minute conference talk from Ray Summit 2025. Discover how Zoox engineers Siddharth Kodwani and Rong Zhou redesigned their ML model-serving infrastructure using Ray Serve to dramatically improve deployment velocity while strengthening system reliability across diverse production workloads. Explore the key challenge of balancing rapid iteration with robust, fault-tolerant deployments and understand how the Zoox team solved this by building a self-service deployment platform powered by dynamic Ray Serve APIs. Examine their architecture featuring dedicated cluster isolation for different use cases, ensuring strong reliability guarantees while enabling fast, independent deployments by engineering teams. Walk through the platform's built-in reliability checks that automatically validate deployments in a self-serve workflow, reducing operations overhead while maintaining system integrity. Understand the seamless integration of LLMs and MLLMs via Ray LLM APIs, which significantly boosted experimentation velocity and made it easy to onboard, benchmark, and iterate on new foundation models. Learn about the platform's support for cost-efficient batch inference as an economical alternative to third-party model APIs. Review key results including automated self-serve deployments that reduced deployment time from days to minutes with built-in reliability guardrails, enhanced reliability through dedicated clusters ensuring performance isolation, accelerated LLM adoption through streamlined onboarding, and a scalable, unified architecture supporting both traditional ML and large-scale LLM workloads in production.
