Video Content Understanding Using Text

This advanced lecture summarizes research on using text to understand video, covering attention-based video representation and query testing, detection and correction of inaccurate content, and text-conditioned frame generation with GANs. It reports experiments on A2D and robotic data and discusses limitations and future work.

Syllabus

Intro
Motivation
Challenges
Algorithm
Training
Video Representation
Scoring function
Optimization - Updating Rules
Exemplar queries
Test on Unseen Queries
Qualitative results
Sentence Encoder
Spatial Attention Network • Which regions of the frames to look?
Temporal Attention Model
Inference Module
Experiments
Limitations
What is an Inaccuracy?
Formulation
Detection By Reconstruction
Visual Features
Inaccuracy Detection
Correction
Last two chapters
How about the opposite problem?
Problem Definition
Proposed Approach - Generator Block Diagram
Text Encoding
Start and End Distributions
Latent Path Construction
Conditional BatchNormalization (CBN)
Frame Generation
UpPooling Block Details
Proposed Approach - Discriminator
Loss Function - Generator
Hinge GAN-Loss on Discriminator
Evaluation Metrics
A2D Quantitative Results
A2D Results
Robotic Results
Dissertation Summary
Future Work