Explore the intersection of differential privacy and large language models in this 10-minute conference talk examining synthetic data generation challenges. Learn how theoretical privacy guarantees can face unexpected real-world complications when document formatting and contextual patterns interact with pre-trained LLMs whose training data remains opaque. Discover the privacy vulnerabilities that can emerge even with conservative epsilon values (ε < 10) and understand the implications for synthetic data applications. Gain insights into the practical limitations of current privacy-preserving synthetic data generation methods when deployed with modern language models, presented by researchers from Carnegie Mellon University at the USENIX Privacy Engineering Practice and Research conference.