Build GenAI Apps from Scratch — UCSB PaCE Certificate Program
Get 20% off all career paths from fullstack to AI
Overview
Google, IBM & Meta Certificates — All 10,000+ Courses at 40% Off
One annual plan covers every course and certificate on Coursera. 40% off for a limited time.
Get Full Access
Learn to build an early-fusion Vision-Language Model from scratch in this comprehensive coding tutorial that demonstrates how these increasingly important models work in robotics and other applications. Master data processing techniques including image resizing to uniform dimensions and text tokenization, then discover how to transform GPT code into a Vision Transformer (ViT) by converting it to an encoder architecture and implementing image patch tokenization. Explore multi-modal model creation by integrating text inputs, implementing causal masking for text tokens, adding padding to increase context size and learning speed, and batching learning data across sub-strings. Gain insights into model training optimization including layer configuration, initialization strategies, and performance tuning, with complete code examples and supplementary slides provided for hands-on implementation.
Syllabus
Micro Vision Langauge Model Coding Tutorial
Taught by
Montreal Robotics