Hacking the Inference Pareto Frontier

Learn advanced techniques for optimizing large language model inference systems through this 20-minute conference talk by Kyle Kranen from NVIDIA. Explore the concept of the Pareto frontier as it applies to AI model deployment, understanding how the trade-offs between cost, throughput, latency, and quality form boundaries that determine where LLM systems can be successfully applied. Discover how models and their inference systems create "token factories" with specific performance characteristics, and examine what happens when systems fall outside optimal trade-off curves. Master cutting-edge optimization techniques enabled by NVIDIA Dynamo, a datacenter-scale distributed inference framework, including disaggregation for separating LLM generation phases, speculation for predicting multiple tokens per cycle, KV routing and storage optimization, and pipelining improvements for agent workflows. Understand the three key drivers for modifying the Pareto frontier: scale, structure, and dynamism, with practical examples of worker specialization and dynamic load balancing. Gain insights into how these advanced inference techniques can reshape performance boundaries, enabling successful deployment of LLM applications that would otherwise be constrained by traditional cost-latency-quality limitations.