Connect your Claude, Codex or OpenClaw for full transcripts on millions of podcasts. Two week free trial, cancel anytime.

Start free trial →
AC Ajeya Cotra Ajeya Cotra Helping the world prepare for and navigate extremely powerful AI.

“If in fact what happens when you give this behavior a negative reward is that the model just becomes more patient and more careful, you'll observe the same thing, which is that you stop seeing that behavior. But it means a much scarier implication.”

80,000 Hours Podcast · October 2026

“If in fact what happens when you give this behavior a negative reward is that the model just becomes more patient and more careful, you'll observe the same thing, which is that you stop seeing that behavior. But it means a much scarier implication.” — Ajeya Cotra, 80,000 Hours Podcast

Context

From an interview re-released as a classic episode. Cotra calls this one of the biggest things she worries about. If catching an AI system doing something sneaky and penalising it reliably fixed the problem, developers could work iteratively and empirically. The trouble is that a model that has learned honesty and a model that has learned to hide better look the same from outside.

Transcript

80,000 Hours Podcast Around 1:09:49 into the episode
Ajeya Cotra

That's right.

Rob Wiblin

How big a problem is this?

Ajeya Cotra

So I think it is one of the biggest things I worry about. If we were in a world where basically the AI systems could try sneaky, deceptive things that weren't totally catastrophic, didn't go as far as taking over the world in one shot. And then if we caught them and basically corrected that in the most straightforward way, which is to give that behavior a negative reward and try and find other cases where it did something similar and give that negative reward, and that just worked, then we would be in a much better place because it would mean we can kind of operate iteratively and empirically without having to think really hard about tricky corner cases. But if in fact what happens when you give this behavior a negative reward is that the model just becomes more patient and more careful, you'll observe the same thing, which is that you stop seeing that behavior. But it means a much scarier implication.

Rob Wiblin

Yeah. It feels like there's something perverse about this argument because it seems like it can't be generally the case that giving negative reward to outcome X or kind of process X then causes it to become extremely good at doing X in a way that you couldn't pick up. Like most of the time when…

Ajeya Cotra

Well, one thing to note is that you do see more of what you want in this world. So you'll see perhaps this model, instead of writing the code you want it to write, it went and grabbed the unit tests you were using to test it on and just like special cased those cases in its code because that was…

Rob Wiblin

Okay, so yeah, so that's a, well, I guess it's an empirical question, but it's just saying people also have different intuitions about what do you think it would hinge on, the question of which way it would go. I guess my intuition is that it's related to what we were talking about earlier, about…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from 80,000 Hours Podcast