“I would argue that a large part of that is how nice natural data is. It's kind of like there's the saying in physics that nature abhors a vacuum. I guess my version of it is that in some sense, nature abhors computationally hard problems.”
Lex Fridman Podcast · Ideas & Essays · October 2026
Moitra on why deep learning works at all, when fitting a neural network to data is cryptographically hard in the worst case. His answer is that natural data, like proteins shaped by evolution, has been selected into problems that are easy to solve. He argues the theory of LLMs needs better models of the world, not just better bounds.
Transcript
Do we have another three hours?
Do we even have intuition? So at the mathematical level, at the philosophical level, somewhere in between all of that, the full range, I don't know how we even answer that.
…very hard. Trying to fit the parameters of a depth for network to data is a cryptographically hard problem. We know that. There exist instances of problems, which, if you could really solve them in a worst-case sense, that would change our understanding of cryptography. And yet, these problems get solved all the time every day when we're fitting deep, you know, much deeper networks than that. So I would argue that a large part of that is how nice natural data is. It's kind of like there's the saying in physics that nature abhors a vacuum. I guess my version of it is that in some sense, nature abhors computationally hard problems. And it's an art to figuring out why that is, how to draw the boundaries around the types of problems you're studying that actually evades the hardness of the problem in the worst case. So there's an interesting example back from the 90s. So certainly back then, and we still are very interested in protein folding. Now, you can model this in a worst case sense. I've got a sequence of amino acids.…
Is it nevertheless amenable to breakthroughs? So, one of the things I would maybe push back on a little bit, just on the pure company incentive side, this is the thing that people talk about, which is the thing that I think OpenAI, Anthropic, all these companies are chasing is the self-improving…
Great question. I mean, I think alignment is one of the most important topics we're facing right now, right? And, you know, just because it's very hard doesn't mean we don't make progress towards understanding it. So, one of the things that I'd worked on recently, we were working on things called…
Yeah, there's, I mean, we've talked about game of life and cellular automata from simple rules emerges this incredible complexity. In that same way, what you've been talking about right now is a single large LLM system. If you take that one system and have it do a lot of reasoning, that is already…
More from Lex Fridman Podcast
