Heterogeneous Memory Opportunity with Agentic AI and Memory Centric Computing

Explore the critical memory bottlenecks limiting large language model (LLM) inference services on GPU systems in this 16-minute conference talk from the Open Compute Project. Learn how limited HBM capacity creates severe constraints on model size, KV cache, and RAG vector databases, resulting in an imbalance with available GPU compute and memory bandwidth resources. Discover emerging second-tier memory solutions including SOCAMM, MRDIMM, and CXL memory as potential solutions to overcome the memory wall, with experimental validation results from DGX GPU systems demonstrating their effectiveness. Examine how CPU-GPU interconnects become the next performance bottleneck and understand Samsung's vision for Processing-Near-Memory (PNM) technology with Processing-in-Memory (PIM) as a fundamental solution to data transfer challenges in memory-centric computing architectures for agentic AI applications.