Coursera Flash Sale
40% Off Coursera Plus for 3 Months!
Grab it
Explore how to compile small language models for edge hardware through advanced AI compiler technology in this 24-minute conference talk. Learn to transform dynamic, unpredictable language model inputs into hardware-optimized workloads by aligning prompt lengths, constraining dynamic shapes, and enabling end-to-end tensor kernel generation. Discover how this approach converts the challenging prefill process from fragmented operations into efficient execution paths that maximize CPU, GPU, and dedicated accelerator utilization. Examine the decoding loop mechanics where single token inputs generate single token outputs while managing growing KV cache, understanding why vector-matrix operations become memory-bound and how attention and cache updates impact latency. Master the critical design decision of KV cache placement as internal model state to unlock layout control, fusion opportunities, and intelligent placement across heterogeneous memory systems. Understand practical techniques for cache partitioning, keeping active layers on accelerators, and managing memory spillover without performance degradation. Review real-world performance gains including 2x to 4x speedups in prefill through input padding with constrained shapes versus per-operation optimizations, with demonstrations showing multi-fold improvements over PyTorch on standard laptops and rapid compilation of recent small models. Address implementation constraints including memory requirements of hundreds of megabytes for quantized models, making this approach suitable for ARM single-board computers, embedded GPUs, and modern laptops while being impractical for ultra-small devices. Gain actionable guidance for implementing fixed prompt length boundaries, propagating constrained shapes through computational graphs, treating KV cache as primary internal resources, and planning heterogeneous execution strategies for fast local generation with enhanced privacy and reliability.
