A Retrieval-based Language Model at Scale - Remote Talk

Explore a remote talk on retrieval-based language models at scale, presented by Sewon Min from UC Berkeley and AI2. Delve into the advantages of retrieval-based LMs as an alternative to dense models, focusing on their ability to combine learned parameters with large datastores. Discover two recent works aimed at improving these models in the context of Large Language Models (LLMs). Learn about a novel pre-training approach for LMs to condition on retrieved documents, and examine the scaling properties of retrieval-based LMs using a massive 1.4 trillion token datastore. Investigate the potential for compute-optional setups across various downstream tasks and consider open-ended questions regarding the impact of retrieval on data training, handling data restrictions, and the possibilities for modular LMs. This 39-minute presentation, part of the "Transformers as a Computational Model" series at the Simons Institute, offers valuable insights into cutting-edge developments in language model technology.