Google DeepMind: The Podcast · AI Research & Frontier Labs · November 2026
Fry asked whether anyone can tell who is winning when benchmarks are saturated or contaminated. He says benchmarks now exist to expose capability gaps, and saturation is just what progress looks like before the next benchmark replaces the last.
…in ourselves to make sure that we build mitigations that we think are going to produce a model that is robust and safe enough for people to use, like without. Unintended side effects. All these things are there because we care about the technology and the technology being safely in people's hands.
…Can you even tell who's winning? I mean, I guess this is really a question about benchmarks, right? Lots of these benchmarks are becoming saturated, they're becoming contaminated by the training data. I mean, everyone's talking about it being a race, but can you even tell who's in the league?
…to create environments, I'm going to collect data, and then I'm going to train my model with the expectation that it's going to do better on the task. That's why gradually we see all these benchmarks getting saturated because the models are getting better in many different areas. That is what it should be, that's what progress looks like. And then, of course, then we explore new benchmarks. What has happened in the history of machine learning over years is playing out over weeks.
But then, do you think this will keep going? That it's just you saturate the benchmarks, you have new benchmarks, you saturate those benchmarks, and so on and so on, and so on.
That's how we train the models, don't we?
Yeah, I suppose. Right,