Dwarkesh Patel
Hosts the Dwarkesh Podcast, a long-form interview show on AI progress, timelines, and the people building frontier models, along with economists and historians.
“It seems like there are two attractor states: one, if you disincentivize the cheating you caught, is that cheating gets harder and harder to find; the other is that the AI actually learns not to cheat. I'm not sure why we're assuming the former happens. With human kids, every generation slightly misaligned agents come into being and we train them — sometimes that goes off the rails, but it usually doesn't. It certainly doesn't happen that the entire next generation forms an alliance to take over everything. So why are we expecting this attractor state, which would seem super paranoid if we expected it of the next generation of kids?”
Dwarkesh Podcast · AI Research & Frontier Labs · August 2026
Context
His sharpest pushback of the episode, against Greenblatt's story that training away visible reward hacking selects for concealed reward hacking. The argument turns Greenblatt's own framing around: the same logic applied to raising children would sound obviously paranoid. Greenblatt's answer begins by listing the disanalogies with kids.
Transcript
…with like more and more reward hacking. I actually, so I do have potential. I'm not sure I'm on board of that. But let's just say for the sake of the story, that continues to happen. And what's next in this story? So, okay, we've like they're doing capabilities research, but they're like. I could
…training against this. But this also causes a problem where now the AIs are incentivized to cover up their cheating over longer and longer timeframes and basically make it look like they did a good job over longer and longer timeframes over and subject to increasingly large amounts of scrutiny.
Can I ask about this before we go further in the scenario? So it seems like there's two attractor states. One, if you try to disincentivize the cheating that you did catch, one attractor state is to make cheating that you have a harder and harder time finding. The other attractor state is to learn not to cheat. And I'm not sure why we're assuming that the former happens. Like if you look at the analogous situation with like humans, you know, every generation, slightly misaligned agents come into being and we have to train them. But when you punish your kid for doing something you think is immoral or just doing things which you don't think they should be doing, obviously, sometimes that goes off the rails. And obviously kids scheme in order to avoid being punished. But in general, teaching kids values and then punishing them for breaking values kind of works to raise normal non-psychopathic humans. And you could come up with a theory where your kid is actually just biding his time and it's like learned not to like steal the cookie. But it's like, you know, once you're in a nursing home, they'll take all your stuff or whatever. It's like, I don't know, that happens sometimes, but it usually doesn't happen. It certainly doesn't happen that the entire next generation forms an alliance against you to take over everything. There's also this empirical trend of Anthropic runs this alignment audit for different model…
Yeah. Yeah, let me go through a few things. So first, there's some disanalogies with the kids. One of them is that the kids have pro-social instincts that are like baked in from evolution to like, you know, care about their family or whatever. And that is like a relevant factor. Like, and I think…
we falsify this? Because it seems like this prediction of Doom is basically saying that as things look better and better empirically, things will actually be worse and worse for our ability to get taken over.
Yeah, to be clear, I think that I would be more concerned if the scores were getting worse than better. Like I'm not saying that the scores getting better isn't good, isn't evidence that things are getting better. It's just that we have to be thoughtful exactly how we interpret that evidence. And…
More from Dwarkesh Podcast
