Introduction to Multimodal Large Language Models I - Day 10 Morning

Explore the fundamentals of multimodal large language models in this comprehensive lecture from JSALT 2025. Learn the core concepts of LLMs including next token prediction, transformers, conformer architectures, and the differences between training and inference processes. Discover supervised fine-tuning (SFT) versus LoRA fine-tuning approaches, and understand how CLIP and CLAP models handle images and audio respectively. Examine the trade-offs between fine-tuning and full training methods, along with essential loss functions including contrastive and cross-attention approaches. Delve into the specifics of training audio language models, covering audio encoders such as CLAP, AST, and Whisper, and explore large audio LLMs including Audio Flamingo-2 and Flamingo-3. Understand dataset creation and synthetic augmentation techniques for audio applications, learn about creating Audio Question Answering (AQA) benchmarks, and explore future directions in audio generation and audio-to-audio processing. Gain insights from leading researchers in multimodal AI who bring expertise from the University of Maryland, Brno University of Technology, and Universidad Autónoma de Madrid, as part of the workshop research group focused on advancing expert-level reasoning and understanding in large audio language models.