Self-driving cars get promised every year and delivered on nobody's schedule. Instead of another hot take, we went through our full library of summarized podcast episodes and pulled the conversations that actually explain why autonomy is hard: the sensors, the edge cases, the human behavior nobody can model cleanly, and the engineers who have to ship something safe anyway.
This is a mix of the people building the cars, the researchers who study why vision and judgment are harder than they look, and a couple of sharp side arguments about whether a computer even needs to be smart to drive one. Expect specifics, not vibes.
Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics | Lex Fridman Podcast #241
Sofman ran Anki before it collapsed in 2019, and now leads Waymo's autonomous trucking effort, which makes him one of the few people who can talk about self-driving from both the toy-robot and the eighteen-wheeler end. He breaks down Waymo Via's use of transfer hubs, the real difference between L4 and L5 autonomy, and why sensor fusion across lidar, camera, and radar exists specifically because no single sensor can be trusted alone. He also makes the case that evaluation, not perception, is the hardest unsolved problem in self-driving. Listen if you want the clearest explanation anywhere of why trucking got automated before robotaxis did.
Read the full episode notesAnca Dragan: Human-Robot Interaction and Reward Engineering | Lex Fridman Podcast #81
Dragan consults at Waymo and treats autonomous driving as fundamentally a game theory problem, not a perception problem: the car has to model what a human driver is about to do, and the human is modeling the car right back. She explains how a self-driving car can nudge into a lane specifically to gather information about another driver's intent, and why semi-autonomous driving, where a human is supposed to take over at a moment's notice, might be the riskier design than full autonomy. This is the episode for anyone who thinks the hard part of self-driving is the cameras.
Read the full episode notesJitendra Malik: Computer Vision | Lex Fridman Podcast #110
Malik has spent a career proving that vision is far harder than it looks precisely because most of what humans do visually happens below conscious awareness. He's openly skeptical of how current self-driving systems handle perception, arguing that feed-forward supervised learning is the wrong foundation and that real systems need to learn the way children do: through exploration and physical interaction with the world, not just labeled datasets. A useful counterweight to industry optimism from someone who helped build the field's underlying tools.
Read the full episode notesMelanie Mitchell: Concepts, Analogies, Common Sense & Future of AI | Lex Fridman Podcast #61
Mitchell uses autonomous driving as her central case study for why she doubts brute-force deep learning gets us to real intelligence. Her argument is the 'long tail': driving isn't hard because of the common cases, it's hard because of the endless rare situations that demand actual common sense, something she says today's systems don't have and can't fake their way into. She also takes on Bostrom and Russell's orthogonality thesis directly. Good listen if you want the skeptic's case laid out by someone who isn't guessing.
Read the full episode notesJim Keller: The Future of Computing, AI, Life, and Consciousness | Lex Fridman Podcast #162
Keller designed chips for Tesla among other companies, and drops a detail that sticks: modern self-driving systems are trained as an arbitrarily large neural network that works, then refactored down into something small enough and cheap enough to actually ship in a car. He also describes a Tesla engineering call to use a tiny onboard computer instead of a resistor to measure battery current, purely because the computer was cheaper. The rest ranges into Moore's Law, consciousness, and a book he's writing about love, but the hardware-meets-the-road segments are the reason to click play.
Read the full episode notesJim Keller: Moore's Law, Microprocessors, and First Principles | Lex Fridman Podcast #70
This earlier conversation has Keller and Lex openly disagreeing about autonomous driving on record: Keller argues 'you don't have to be especially smart to drive a car,' framing autonomy as mostly an attention problem that computers are already built to win, while Lex pushes back on how much human behavior actually matters. Along the way Keller explains modern branch predictors, which now work something like small neural networks hitting 99% accuracy. The disagreement itself is the reveal here.
Read the full episode notesMIT 6.S094: Computer Vision
This is Lex Fridman's own MIT lecture, not an interview, and it builds computer vision from a trivial pixel-difference classifier up through the ImageNet-winning architectures (AlexNet, ResNet, and the rest) before landing on the part that matters for this list: the SegFuse competition, where students use optical flow and temporal information to improve frame-by-frame segmentation of actual driving scenes toward ground truth. If you want to understand what 'the car sees the road' technically means, this is the most direct explanation on the list.
Read the full episode notesYann LeCun: Deep Learning, ConvNets, and Self-Supervised Learning | Lex Fridman Podcast #36
LeCun makes a comparison that reframes the whole self-driving problem: today's reinforcement learning would need millions of driving hours, and would kill thousands of virtual pedestrians and run off virtual cliffs along the way, while humans learn to drive competently in 20 to 30 hours because they already have a world model to build on. His broader argument, that machines need self-supervised learning to build common sense the way babies do, is really an argument about why driving is a bad fit for the RL approach everyone defaults to.
Read the full episode notesEssentials: Machines, Creativity & Love | Dr. Lex Fridman
In this Huberman Lab Essentials cut, Lex frames Tesla Autopilot as one of AI's highest-stakes real-world deployments precisely because human lives are on the line every time the system takes a decision. He explains Karpathy's 'data engine' concept, where Tesla's retraining loop pulls edge cases from hundreds of thousands of cars on the road, as a live example of the messy human-and-machine collaboration he thinks is more interesting than full autonomy. The conversation drifts into robot companionship and grief, but the Autopilot framing is worth the listen on its own.
Read the full episode notesDawn Song: Adversarial Machine Learning and Computer Security | Lex Fridman Podcast #95
Song's lab built adversarial stop signs, physical stickers that reliably fool autonomous-driving image classifiers, robust enough that one now sits in a London science museum as an exhibit. She says she's '100 percent confident' that physical adversarial attacks on Tesla's systems are feasible, directly contradicting Elon Musk's public dismissal of the risk. This is the episode for anyone who assumes a self-driving car's biggest vulnerability is a software bug rather than a sticker someone put on a sign.
Read the full episode notesThat's ten conversations that cover the entire stack, from the sensors and the chips underneath them to the philosophical argument over whether a machine can ever really drive like a human does. Browse the full library of episode summaries on Episode Notes for more, whether you're chasing autonomous driving specifically or just want to know which episode is worth your next commute.