Leveraging Structure in Formal Logic-Guided Reinforcement Learning

Explore a 51-minute lecture by Ameesh Shah from UC Berkeley that delves into formal logic-guided reinforcement learning approaches. Learn how temporal logic and automata can serve as alternatives to traditional Markovian rewards in specifying objectives for deep reinforcement learning agents. Discover the advantages of formal specifications, including their composability, transferability across environments, and precise satisfaction metrics. The talk examines how to combine Markovian rewards with formal specifications to create policies that optimize rewards while maximizing the probability of satisfying formal constraints. Shah addresses reward sparsity challenges in deep learning contexts and introduces an innovative approach to multi-agent reinforcement learning that decomposes formal task specifications into individual sub-tasks for more efficient cooperative objective solving. The presentation concludes with insights on how symbolic and formal task structures can enhance future agentic learning systems, as part of the Theoretical Aspects of Trustworthy AI series at the Simons Institute.