PUMICE - A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations

Explore a 20-minute conference talk from the ACM Symposium on User Interface Software and Technology that introduces PUMICE, a multi-modal agent designed to learn concepts and conditionals through natural language and demonstrations. Discover how this innovative approach combines natural language programming with programming-by-demonstration, allowing users to describe tasks and conditions at a high level before collaborating with the agent to resolve ambiguities. Learn about the system's ability to define new procedures and concepts using existing mobile app GUIs, and understand its potential impact on end-user programming. Gain insights into the talk's structure, covering introduction, demonstration, conversation structures, questions, limitations, and tradeoffs of this domain-independent approach.