How Coinbase Uses Ray, vLLM and LiteLLM to Power Secure LLM Services

Learn how to build trusted, production-grade LLM services in security-sensitive financial environments through this 16-minute conference talk from Ray Summit 2025. Discover Coinbase's approach to engineering LLM infrastructure that meets the non-negotiable requirements of trust, security, and reliability in one of the world's most security-conscious crypto exchanges. Explore the technical architecture behind Coinbase's internal LLM services, including user authentication and authorization patterns tailored for secure LLM access, service-to-service trust models for safe and auditable communication between internal systems, and LiteLLM distribution strategies to balance throughput, reliability, and fallback behavior. Understand how Ray enables distributed orchestration and scaling, how vLLM delivers high-throughput, low-latency inference, and how LiteLLM provides routing, abstraction, and multi-provider reliability in the integrated serving stack. Examine the systems built to support high-volume internal LLM traffic while ensuring consistent performance under load, and gain insights into the full end-to-end implementation of Ray and vLLM for delivering trustworthy, secure, and efficient LLM services that meet the strict reliability requirements of a top global crypto exchange.