Dataset Inference Methods for Large Language Models - Detecting Training Data Usage

Learn about groundbreaking research on dataset inference methods for large language models in this technical talk from Google TechTalks. Explore how traditional membership inference attacks (MIAs) face limitations when identifying training data, and discover a novel approach to accurately determine if specific datasets were used in model training. Delve into the challenges of distribution shifts in membership inference, understand why many MIA methods perform similarly to random guessing when comparing members and non-members from the same distribution, and examine a new statistical testing framework that combines selective MIAs for reliable dataset-level inference. Follow along as Carnegie Mellon University researcher Pratyush Maini demonstrates how this innovative method successfully distinguishes between training and test sets from the Pile dataset with statistical significance, addressing real-world copyright concerns in the era of large language models.