Show notes
Deep learning works extraordinarily well. And we still largely don't know why.A new paper from Jamie Simon, Daniel Kunin, and 12 co-authors argues that a scientific theory of deep learning is emerging, and coins a name for the emerging field: learning mechanics.We sat down with Jamie and Dan on Generally Intelligent to talk about what a physics of deep learning would actually look like, why now, and what's left to figure out.00:03:05 Learning mechanics as the physics to mechanistic interpretability's biology00:04:13 Why deep learning needs a theory00:07:07 Why deep learning is uniquely hard to engineer00:12:11 How a week in the woods became a paper00:25:59 The barrier to theory isn't opacity, but complexity00:36:26 Deep learning's first gas law00:47:22 Why more particles makes the problem easier 00:56:22 The discretization hypothesis01:01:50 The strongest signal that a compact theory exists01:05:07 The Platonic Representation Hypothesis01:15:41 Why learning mechanics and mech interp need each other01:25:29 Theory as safety infrastructureRead the paperTranscript and linksLearning Mechanics website Full transcript: https://imbueai.substack.com/p/geoffrey-littGenerally Intelligent is a podcast by Imbue, a research company building toward a future where AI agents are open and accountable to their users, so people have more power in the digital world.WebsiteSubstackXLinkedInYouTube