Evaluating Data Misuse in LLMs - Introducing Adversarial Compression Rate as a Metric of Memorization

Learn about a novel metric for assessing data memorization in large language models through this Google TechTalk presentation. Explore the Adversarial Compression Ratio (ACR), a new approach to determining whether LLMs memorize training data or synthesize information more like human learning. Discover how this metric works by measuring whether training data strings can be elicited by prompts significantly shorter than the original content, effectively "compressing" the data through adversarial prompts. Understand the advantages of ACR over existing memorization measures, including its adversarial perspective for monitoring unlearning and compliance, and its computational efficiency for evaluating arbitrary strings. Examine the practical applications of this definition as a tool for identifying potential violations of data usage terms and its implications for legal frameworks surrounding LLM training practices. Gain insights into the broader questions of permissible data usage in web-scale model training and the technical approaches to addressing privacy and compliance concerns in machine learning systems.