Neuro-Symbolic AI for Scene Understanding in Autonomous Systems

Attend a Ph.D. defense presentation exploring the integration of neurosymbolic AI approaches for advancing scene understanding capabilities in autonomous systems. Discover how combining knowledge representation, representation learning, and reasoning can address critical limitations in safety, robustness, generalization, and explainability that current computer vision and deep learning models face when operating solely on raw sensor data. Learn about the construction of unified knowledge representations that integrate scene data with background knowledge, including the development of dataset-agnostic scene ontologies and knowledge graphs for representing multimodal autonomous system data. Explore DSceneKG, a suite of large-scale knowledge graphs representing real-world driving scenes across multiple autonomous driving datasets, and understand its applications in explainable scene clustering, causal reasoning, and industrial cross-modal retrieval tasks. Examine methods for enhancing the expressiveness of scene knowledge in sub-symbolic representations to support downstream learning tasks through effective knowledge graph patterns and structures that improve semantic richness and model reasoning capabilities. Investigate knowledge-based entity prediction (KEP), a novel cognitive visual reasoning task that leverages relational knowledge to predict entities not directly observed but likely to exist given scene context. Understand CLUE, a context-based method for labeling unobserved entities designed to improve annotation quality in multimodal datasets by incorporating contextual knowledge of potentially missing entities due to perceptual failures. Review CUEBench, a comprehensive benchmark for contextual entity prediction that systematically evaluates both neurosymbolic and foundation model-based approaches including large language models and multimodal language models, addressing the critical gap in benchmarking high-level cognitive reasoning under perceptual incompleteness in real-world autonomous system scenarios.