“…benchmarks become the environment that these models are evolving inside of.”
The Analytics Engineering Podcast · Hands-On Engineering Podcasts · August 2026
Talking about whether AI models will converge or diverge over time, Handy reaches for evolution. Benchmarks, he argues, act like the predators and food scarcity that shaped humans — the environment a model adapts to — so models pushed against the same benchmarks will tend to converge on the same behavior.
Transcript
for working that in.
…like, it's a bit of an open question to me on how much, how much we'll actually see that. And then therefore, how much true kind of fluidity in the at the highest layers. There's always going to be some set of workloads that are addressable within this. It's just like a question of what percentage.
…router represents. But I agree that's a long-term statement and it's not clear. Over time, I think that the evolutionary, we were talking about benchmarks right before this. And benchmarks are, you can think of them almost as like evolutionary pressure. How did humans evolve in environments where there were large predators or a scarcity of food or a scarcity of what? Like these are, you know, benchmarks become the environment that these models are evolving inside of. And to the extent that all the models are. Responding to the same evolutionary pressure, then they will converge on some time horizon. But the other case, I guess, is that you have agents that are very explicitly, or sorry, models that are very explicitly tuned to be better in some evolutionary conditions versus others. Maybe that trade-off makes sense. And there's like the class of model that…
That's right. And if you're a data analyst at Stripe and you want to come on and talk to Tristan Ley about tokenomics, let us know.
Where are we going from here?
Okay. So I think a perfect way to close this out is, you know, in the benchmarking section, we were talking about the fact that models have gotten pretty good at the set of the things that we are currently addressing within data orgs. And like the fundamentals are still very important and you still…
More from The Analytics Engineering Podcast
