Efficient Computing for Deep Learning, Robotics, and AI (Vivienne Sze) | MIT Deep Learning Series

In this MIT Deep Learning Series lecture hosted by Lex Fridman, Vivienne Sze gives a broad overview of efficient computing for deep learning, robotics, and AI. She explains that the explosive growth in compute demand and its carbon footprint, combined with the slowdown of Moore's Law and Dennard scaling, forces a turn toward specialized hardware. Her central theme is that power is dominated by data movement rather than computation, so the key to efficiency is reducing how often and how far data travels. She walks through hardware accelerators (the Eyeriss and Navion chips), algorithmic techniques like pruning and reduced precision, and joint hardware-algorithm co-design tools like NetAdapt. She closes with applications beyond neural nets, including robot localization, super-resolution, and low-cost in-home monitoring of neurodegenerative disease via eye movements on a phone.

• Compute required for deep learning grew exponentially, by over 300,000 times, to drive accuracy gains in recent years.
  00:01:34
• The key contrarian insight: power is dominated by data movement, not the computations themselves.
  00:09:20
• Reading data from off-chip DRAM costs two orders of magnitude more energy than performing the actual multiply-and-accumulate.
  00:21:47
• AlexNet and VGG cost two to three orders of magnitude more energy than HOG features for object detection.
  00:36:20
• Energy-aware pruning (removing high-energy weights first) yields a 3.7x energy reduction versus 2x for traditional magnitude-based pruning at the same accuracy.
  00:42:33
• Their FastDepth approach runs real-time monocular depth estimation at about 40 frames per second on an iPhone.
  00:51:23
• Eye-reaction-time measurements on a sub-$1,000 iPhone 6 matched the distributions of an expensive phantom camera, enabling low-cost in-home disease monitoring.
  01:13:08

• Training a neural net can have a carbon footprint orders of magnitude greater than flying across North America or an average human life.
  00:02:06
• A self-driving car can consume over 2000 watts just for computation, generating heat that often needs water cooling.
  00:04:10
• Handheld devices are typically limited to under a watt of power due to heat dissipation constraints.
  00:05:11
• The multiply-and-accumulate operation accounts for over 90% of the computation in neural networks.
  00:14:02
• Every multiply-and-accumulate requires four memory accesses, a four-to-one ratio of memory access to compute.
  00:21:17
• Skipping multiplications where an input is zero can reduce power consumption by almost 50%.
  00:31:08
• HOG features are actually more energy-efficient than video compression for understanding a pixel.
  00:36:20
• Their hardware can compute mutual information over a 200m x 200m map in under a second, a 100x speedup over a CPU.
  01:10:02
• Their depth-sensing method turns the power-hungry depth sensor on only about 11% of the time with only ~0.7% mean relative error.
  01:14:42

Books, products and media the guest or host genuinely endorsed here — with the buy link.

Affiliate link — we may earn a commission at no extra cost to you.