Lex Fridman — MIT researcher and lecturer teaching the 6.S094 deep learning course; presents computer vision concepts and the SegFuse dynamic scene segmentation competition.
The gist
This is a solo MIT lecture on computer vision as it stands today, framed almost entirely around deep learning. Fridman builds intuition from a trivial pixel-difference classifier and K-nearest neighbors up through convolutional neural networks, walking through the convolution operation, filters, stride, padding, and pooling. He surveys the lineage of ImageNet-winning architectures (AlexNet, VGGNet, GoogLeNet/Inception, ResNet, and SENet), highlighting the key insight each introduced. The second half moves to fully convolutional networks for pixel-level semantic scene segmentation, encoder-decoder frameworks, dilated convolutions, and conditional random fields, then to optical flow via FlowNet. It culminates in the SegFuse competition, which challenges students to use temporal information and optical flow to improve frame-by-frame driving-scene segmentation toward ground truth.
Big reveals
- A trivial image-difference classifier hits ~35-38% on CIFAR-10, far above the 10% random-chance baseline.
- On CIFAR-10, K-nearest neighbors reaches ~30%, human level is 95%, and CNNs get close to 100%.
- ResNet surpassed human-level ImageNet performance (5.1% error) in 2015, achieving ~4% top-5 error.
- Squeeze-and-Excitation Networks won 2017 by adding a learnable per-channel weighting, cutting error roughly 25% (4% to 3%).
- Capsule networks expose that convolutional nets throw away spatial-relationship/pose information, so two scrambled faces look identical to a CNN.
- Manually segmenting a single Cityscapes-style image (coloring every pixel) takes about 90 minutes, which is why no dense video-segmentation dataset exists yet.
- FlowNet 2.0's accuracy depended heavily on the ORDER in which the training datasets were presented during training.
Things worth remembering
- In supervised vision, humans provide annotated ground-truth labels and the network learns to map raw sensory input to those labels, then generalize.
- Deep neural networks for vision are inspired by the layered visual cortex, where higher-order representations form as info passes from the eyes deeper into the brain.
- Illumination variability and occlusion are cited as two of the biggest challenges in driving perception with visible-light cameras.
- CNNs exploit spatial invariance: a cat in the top-left corner is the same as a cat in the bottom-right, so features are shared across the image to save parameters.
- ImageNet contains 14 million images across 21,000 categories, with depth like 1,200 Granny Smith apple images.
- VGGNet (2014) used a uniform conv-pool architecture with 138 million parameters and few optimizations.
- GoogLeNet's Inception module runs 1x1, 3x3, and 5x5 convolutions together rather than choosing one size, lowering parameter count.
- Optical flow estimates the direction and magnitude each pixel moved between two frames 33.3 milliseconds apart at 30 fps.
- The SegFuse dataset includes original 1080p HD and 8K 360 driving video shot around Cambridge, with frame-by-frame ground-truth segmentation on Mechanical Turk.
- SegFuse provides 10,000 images with Python starter code on GitHub for the competition.