POPri - Private Federated Learning using Preference-Optimized Synthetic Data

Learn about POPri (Policy Optimization for Private Data), a novel approach to private federated learning that leverages preference-optimized synthetic data generation in this Google TechTalk. Discover how this method addresses privacy concerns in machine learning by using differentially private synthetic data instead of traditional DP-FL approaches. Explore the key insight that private client feedback can be treated as reinforcement learning rewards, enabling the use of policy optimization algorithms like Direct Preference Optimization (DPO) to fine-tune large language models for generating high-quality differentially private synthetic data. Examine the methodology behind POPri's approach to collecting and utilizing client feedback without compromising privacy, and understand how it eliminates the need for careful prompt engineering based on public information. Investigate the performance evaluation conducted on LargeFedBench, a newly released federated text benchmark designed for uncontaminated LLM evaluations on federated client data. Analyze the substantial improvements POPri demonstrates over prior synthetic data methods, including closing the gap between next-token prediction accuracy in fully-private and non-private settings by up to 58%, compared to 28% for previous synthetic data approaches and only 3% for state-of-the-art DP federated learning methods. Gain insights into the practical implications of this research for privacy-preserving machine learning applications and the future of federated learning systems.