Explore the concept of Streaming Language Models with Attention Sinks in this informative video explanation. Delve into the challenges of deploying Large Language Models (LLMs) for streaming applications and long interactions. Learn about the phenomenon of attention sinks and how they can recover the performance of window attention. Discover the StreamingLLM framework, which enables LLMs to generalize to infinite sequence lengths without fine-tuning. Examine experimental evidence, explore the semantics versus position debate, and investigate whether attention sinks can be learned. Compare this approach to other models like Big Bird and gain insights into efficient language modeling with extended token capabilities.

Syllabus

- Introduction
- What is the problem?
- The hypothesis: Attention Sinks
- Experimental evidence
- Streaming LLMs
- Semantics or position?
- Can attention sinks be learned?
- More experiments
- Comparison to Big Bird