Google, IBM & Meta Certificates — All 10,000+ Courses at 40% Off
One annual plan covers every course and certificate on Coursera. 40% off for a limited time.
Get Full Access
Explore a groundbreaking model serving system called SHEPHERD in this 15-minute conference talk from NSDI '23. Discover how SHEPHERD tackles the challenges of scalability, high system goodput, and maximum resource utilization across compute units for inference requests in interactive web services. Learn about its innovative two-level design that separates planning and serving modules, leveraging request stream aggregation for improved predictability and resource utilization. Understand the novel online algorithm employed by SHEPHERD for guaranteed goodput under unpredictable workloads, utilizing preemptions and model-specific batching properties. Gain insights into the system's performance, which achieves up to 18.1X higher goodput and 1.8X better utilization compared to prior state-of-the-art solutions, while scaling to hundreds of workers.
