Natural Signals Properties and the Convolution

This lecture explains how natural signal properties such as locality and stationarity motivate convolutional neural networks. It covers sparsity, parameter sharing, kernels, padding, pooling, tensor reshaping, and a notebook comparison with fully connected networks.

Syllabus

– Happy birthday to the TAs!
– Today topic: convolutional neural nets
– Input layer, points, and signals
– Natural signal properties
– 1D stationarity
– 1D locality
– 2D stationarity
– 2D locality
– 2D compositionality
– Fully connected recap
– Locality ⇒ sparsity
– Stationarity ⇒ parameter sharing
– 1D kernels
– 1D padding
– ConvNet for images and tensor reshaping
– Pooling
– Jupyter Notebook: fully connected vs. convnet
– Deterministic pixel shuffling: breaking signal properties
– Final comparison
– Goodbye