Things Ilya Sutskever Has Said on Podcasts

“The year was 2003. Today, we take it for granted that computers can learn. But in 2003, we took it for granted that computers can't learn. The biggest achievement of AI back then was Deep Blue, the chess playing engine.” — Ilya Sutskever, Eye On A.I.

Sutskever was recalling how he started working with Geoff Hinton at the University of Toronto at 17, when machine learning seemed "completely unaccessible" and learning was a big mystery. The interview was recorded while he was at OpenAI and re-released by Eye on AI.

Eye On A.I. · 2026-10-09 Permalink → Listen →

Eye On A.I. Around 00:58 into the episode

Speaker 1

…More broadly, progress has gone increasingly vertical. Specialized domain-specific models in areas like cybersecurity, law, health, and media are growing faster than general chatbots and often beat larger foundation models on their home turf. With all that as backdrop, here's our conversation.

Speaker 2

…were some of the ways that got me started. And I actually started working with Jeff Hinton very early when I was 17 because we moved to Canada and I immediately was able to join the University of Toronto. And I really wanted to do machine learning because that seemed like the most important aspect of artificial intelligence that at the time was completely unaccessible. Like to give some context, the year was 2003. Today, we take it for granted that computers can learn. But in 2003, we took it for granted that computers can't learn. The biggest achievement of AI back then was deep blue, the chess playing engine. But there it was like you have this game and you have this research and you have this simple way of determining if one position is better than another. And it really did not feel like that could possibly be applicable to the real world because there is no learning and learning was this big mystery. And so I was really, really interested in learning. And to my great luck, Jeff Hinton was a…

Speaker 3

And was your impulse as it was for Jeff to understand how the brain worked or was it more that you were simply interested in the idea of machines learning?

Speaker 2

AI is so big. And so the motivations were just as many. Like it is interesting, but how does intelligence work at all? Like right now, we have quite a bit of an idea that it's a big neural net and we know how it works to some degree. But back then, although the neural nets were around, no one knew…

Speaker 3

And then Alex, and I've spoken to Jeff and he said that it was really your excitement about the breakthroughs in convolutional neural networks that led you to apply for the ImageNet competition and that Alex had the coding skills to train the network. Can you talk just a little bit about that? I…

Speaker names from our own diarization · position estimated from where the line sits in the episode

“I actually think that the bitter lesson, as articulated, overstates its case, or at least I think the takeaway that people have taken from it overstates its case. The takeaway that people have is: doesn't matter what you do, just scale. But that's not exactly true. You got to scale something specific. You got to have something that will be able to benefit from the scale.” — Ilya Sutskever, Eye On A.I.

Asked whether Rich Sutton's essay "The Bitter Lesson" had influenced him, Sutskever said deep learning was the first thing that was worth scaling, and that some twist on it might one day scale better. Eye on AI recorded the interview while he was still chief scientist at OpenAI and re-released it this week.

Eye On A.I. · 2026-10-09 Permalink → Listen →

Eye On A.I. Around 09:43 into the episode

Speaker 2

…right away. And so the very nascent GPT effort continued then. And then, like with the transformer, it started to work better and you make it bigger. And then we realized we need to keep making it bigger. And we did. And that's what led to eventually GPT-3 and essentially where we are today.

Speaker 3

…to the problems or the shortcomings of large language models or large models generally. But Rich Sutton had been writing about scaling and how that's all we need to do. We don't need new algorithms. We just need to scale. Did he have an influence on you? Or was that a parallel track of thinking?

Speaker 2

No. I would say that when he posted his article, then we were very pleased to see. Some external people thinking in similar lines, and we thought it was very eloquently articulated. But I actually think that the bitter lesson, as articulated, overstates its case, or at least I think the takeaway that people have taken from it overstates its case. The takeaway that people have is: doesn't matter what you do, just scale. But that's not exactly true. You got to scale something specific. You got to have something that will be able to benefit from the scale. The great breakthrough of deep learning is that it provides us with the first ever way of productively using scale and getting something out of it in return. Like before that, like what would people use large computer clusters for? I guess they would do it for weather simulations or physics simulations or something, but that's about it, maybe movie making. But no one had any real need for…

Speaker 3

The limitation of large language models as they exist is their knowledge is contained in the language that they're trained on. And most human knowledge, I think everyone agrees, is non-linguistic. I'm not sure Noam Chomsky agrees, but there's a problem in the large language models. As I understand…

Speaker 2

Yeah. So before I comment on the immediate question that you ask, I want to comment about some of the earlier parts of the question.

Speaker 3

Sure.

Speaker names from our own diarization · position estimated from where the line sits in the episode

Things Ilya Says on Podcasts

Where to Find Them

What They Said

Collections They Appear In