GenAI Traffic: Why API Infrastructure Must Evolve Again

Join this MLOps Podcast episode featuring Erica Hughberg, Community Advocate at Tetrate, as she discusses why API infrastructure needs to evolve to handle GenAI traffic. Learn how traditional API gateways optimized for microservices are struggling with the new demands of generative AI workloads, which require long-lived connections, massive payloads, and streaming responses. Explore the evolution from the C10K problem through the microservices revolution to today's AI-driven challenges that demand new approaches to token-based rate limiting, cost-aware request shaping, and scalable AI inference traffic. Hughberg, a technical leader and maintainer of Envoy AI Gateway, shares insights on building scalable, secure application platforms while making complex technical concepts accessible through engaging narratives.