Scaling GenAI Inference - Techniques, Optimizations, and Real-World Lessons

Learn practical strategies for scaling generative AI models from research prototypes to production-grade systems in this 16-minute conference talk. Discover how to overcome core obstacles in GenAI inference while reducing latency and controlling costs without sacrificing model performance. Explore advanced optimization techniques including batching strategies, model quantization methods, parallelism approaches, KV cache management, and speculative decoding implementations. Gain insights from real-world experience as the session unpacks critical trade-offs, common pitfalls, and essential lessons learned from successfully scaling inference systems in production environments.