Robot Learning: Learning to Plan with Large Models - Part 2

Watch a 49-minute technical lecture from Montreal Robotics exploring latent inverse models and their applications in video generation and robotics. Delve into how these models develop predictive capabilities by embedding world dynamics understanding, particularly for complex scenarios involving human motion. Learn about conditioning predictions on latent action models for controlled manipulation, the importance of task-specific codebooks in defining actions, and the critical role of data quality and image fidelity in the learning process. Examine the evolution from traditional inverse dynamics models to this integrated action control approach, with special attention to the "Genie" project that demonstrates how scaling affects the capture of detailed dynamics in robotics applications. Gain insights into the training methodology of latent dynamics models and their ability to infer underlying states from image data.