A General Framework for Inference-time Scaling and Steering of Diffusion Models

This talk explores a novel approach called Feynman Kac (FK) steering, a framework for controlling diffusion models at inference time without expensive retraining. Learn how this method works by sampling multiple interacting diffusion processes (particles) and resampling them based on potential functions derived from reward signals. The presentation covers various potential choices, intermediate rewards, and sampling techniques, demonstrating impressive results across text-to-image and text diffusion models. Discover how FK steering a smaller 0.8B parameter model outperforms a 2.6B fine-tuned model on prompt fidelity while offering faster sampling with no training requirements. The talk also shows how this approach generates higher quality text outputs with better linguistic acceptability and enables gradient-free control of text attributes like toxicity. Access the implementation code on GitHub and connect with the speakers through the Valence Labs Portal community for AI drug discovery.