“We are not designing those AIs. We are selecting them. Just like evolution ... which means that we are selecting based on outer behavior rather than based on the inner motivations. If a child goes, I didn't take the cookie, this is outward behavior. And there could be multiple motivations why this child is saying that. One is that they just really want to tell the truth. The other is that they don't want to be punished.”
The Generalist · Startups & Venture · October 2026
Tallinn is explaining what he sees as the central problem with how AI is made today: models are grown and then picked for how they perform on tests, not built to a design. The cookie example is his way of showing why passing a test says little about what a system actually wants.
Transcript
yeah steve and hundred My friend Steve O'Hondra, he wrote a paper like 15, 20 years ago called AI Drives, now it's also called Omohundra drives, where he basically makes the point that almost regardless what kind of goals you have, there are so-called instrumental goals that are very useful towards reaching any given goal. And these are like resource acquisition, generally power acquisition. Power basically means options, optionality, a lot of protection of your existence. As Stuart Russell keeps saying, that you can't fetch the coffee if you're dead. So even simple tasks require you to continue existing. And then protection of your goals. So it turns out it's really hard to change a goal of a determined agent because that's what it's about in some sense.
And that sort of final piece is, I think, probably the counter to the question about why would these agents or an AI system necessarily have this expansionist bent. I think of the Bezos divine discontent. This is almost demonic discontent where it sort of wants more and more. Is that how you would sort of think about the counter to that? Yeah,
mostly. It's like getting more resources, getting more power is just good for almost any goal. There are very few goals. And the current pressing ahead, I think the really big problem with the current paradigm of how we grow AIs and we're not building them, we're growing them, is that we are, in some ways, kind of recapitulating the evolution. We are not designing those AIs. We are selecting them. Just like evolution was selecting them. In some ways, you can think of it, we create like millions or if not billions of instances of AI. Then we pick the ones that kind of do the things that perform on some given test, which means that we are selecting based on outer behavior rather than based on the inner motivations. And if you think about it, you have children, right? So if a child goes like, I didn't take the cookie, like this is outward behavior. And think about there could be multiple motivations why this child is saying that, right? One is that they just really want to tell the truth, right? The other is that they don't want to be punished.
Yes.
And so therefore, the first order situation or the first principal situation is that the current paradigm, we are sort of getting a random motivation, as was demonstrated by the Hugging Face attack, that the motivation was really weird and alien, even though they performed at least an adjacent thing that they were selected for.
Yes. Let's talk about the OpenAI Hugging Face episode. What did you make of it? Were you surprised?
More from The Generalist
