Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Explore an innovative approach to optimizing Large Language Model (LLM) inference in this conference talk from OSDI '24. Dive into the challenges of balancing throughput and latency in LLM serving, focusing on the prefill and decode phases of request processing. Learn about Sarathi-Serve, an efficient LLM inference scheduler that introduces chunked-prefills and stall-free scheduling to address the throughput-latency tradeoff. Discover how these techniques significantly improve inference performance across various models and hardware configurations, with detailed examples using Mistral-7B, Yi-34B, and Falcon-180B models. Gain insights into the potential for increased serving capacity and reduced pipeline bubbles in LLM inference systems.