Rethinking and Improving Relative Position Encoding for Vision Transformer - Lecture 23

This lecture examines methods for improving relative position encoding in Vision Transformers. It covers 2D relative position calculation, contextual and directed bias, piecewise indexing, implementation details, experiments, and complexity analysis.

Syllabus

Intro
Background and previous work
Self-attention
Absolute Position Encoding and Relative Position Encoding (RPE)
RPE in Transformer-XL
Bias and Contextual Mode
A Piecewise Index Function
2D Relative Position Calculation
Experiments
Implementation details
Directed vs. Undirected Bias vs. Contextual
Shared v.s. Unshared
Piecewise v.s. Clip
Number of buckets
Component-wise analysis
Complexity Analysis
Visualization
Conclusion