AI, Data Science & Cloud Certificates from Google, IBM & Meta — 40% Off
One plan covers every Professional Certificate on Coursera. 40% off Coursera Plus Annual.
Unlock All Certificates
Explore a groundbreaking development framework for complex multimodal interactions using large language models in this conference talk from CHI 2024. Delve into ReactGenie, which combines voice and touch interactions to create more efficient multimodal interfaces. Learn how this framework simplifies the development process by separating multimodal input from the computational model, enabling easier creation of capable interfaces. Discover the use of NLPL (Natural Language Programming Language) and neural semantic parsing to translate user commands. Examine the framework's ability to interpret parsed NLPL and compose primitives for implementing complex user commands. Gain insights from an evaluation showing developers can learn and build non-trivial ReactGenie applications quickly, and how end-users complete tasks faster with less cognitive load compared to traditional GUIs.
