Build Motion Control Interfaces with Multimodal LLMs

Explore how to create motion-controlled interfaces using multimodal Large Language Models in this 29-minute conference talk from JSNation 2025. Discover the intersection of multimodal AI and human-computer interaction as Charlie Gerard demonstrates practical applications for controlling websites and IoT devices through gesture recognition. Learn to implement TensorFlow.js models for motion detection and enhance user experiences with gesture-based controls that require minimal traditional input methods. Examine how LLMs can interpret and respond to physical movements, with hands-on examples using Gemini for webcam-based gesture recognition. Follow along as the speaker demonstrates controlling WiFi light bulbs through hand gestures, implementing multi-device control systems with positional awareness, and creating custom gesture databases for personalized interfaces. Understand the integration of TensorFlow.js for color control applications and explore the potential for autonomous interfaces that respond to natural human movements. Gain insights into redefining coding interfaces through motion control, leveraging live API functions for directional control, and building custom gesture recognition systems that can be tailored to individual users and specific use cases.