“I actually think that the bitter lesson, as articulated, overstates its case, or at least I think the takeaway that people have taken from it overstates its case. The takeaway that people have is: doesn't matter what you do, just scale. But that's not exactly true. You got to scale something specific. You got to have something that will be able to benefit from the scale.”
Eye On A.I. · October 2026
Asked whether Rich Sutton's essay "The Bitter Lesson" had influenced him, Sutskever said deep learning was the first thing that was worth scaling, and that some twist on it might one day scale better. Eye on AI recorded the interview while he was still chief scientist at OpenAI and re-released it this week.
Transcript
…right away. And so the very nascent GPT effort continued then. And then, like with the transformer, it started to work better and you make it bigger. And then we realized we need to keep making it bigger. And we did. And that's what led to eventually GPT-3 and essentially where we are today.
…to the problems or the shortcomings of large language models or large models generally. But Rich Sutton had been writing about scaling and how that's all we need to do. We don't need new algorithms. We just need to scale. Did he have an influence on you? Or was that a parallel track of thinking?
No. I would say that when he posted his article, then we were very pleased to see. Some external people thinking in similar lines, and we thought it was very eloquently articulated. But I actually think that the bitter lesson, as articulated, overstates its case, or at least I think the takeaway that people have taken from it overstates its case. The takeaway that people have is: doesn't matter what you do, just scale. But that's not exactly true. You got to scale something specific. You got to have something that will be able to benefit from the scale. The great breakthrough of deep learning is that it provides us with the first ever way of productively using scale and getting something out of it in return. Like before that, like what would people use large computer clusters for? I guess they would do it for weather simulations or physics simulations or something, but that's about it, maybe movie making. But no one had any real need for…
The limitation of large language models as they exist is their knowledge is contained in the language that they're trained on. And most human knowledge, I think everyone agrees, is non-linguistic. I'm not sure Noam Chomsky agrees, but there's a problem in the large language models. As I understand…
Yeah. So before I comment on the immediate question that you ask, I want to comment about some of the earlier parts of the question.
Sure.
More from Eye On A.I.
