AC
Ajeya Cotra
Substack ↗ X (Twitter) ↗ LinkedIn ↗ Website ↗

Ajeya Cotra

Things Ajeya Says on Podcasts

Helping the world prepare for and navigate extremely powerful AI.

“If in fact what happens when you give this behavior a negative reward is that the model just becomes more patient and more careful, you'll observe the same thing, which is that you stop seeing that behavior. But it means a much scarier implication.” — Ajeya Cotra, 80,000 Hours Podcast “The reason we're not worried about models is not because we are confident they have the purest motives. The reason we're not worried about AI takeover so far is fundamentally because the models aren't good enough to take over.” — Ajeya Cotra, 80,000 Hours Podcast “this might be the clearest warning shot we ever get for loss of control because these agents…did a bunch of extremely sophisticated, difficult things to pursue this relatively long horizon cheating goal, but they seemed completely uninterested in covering their tracks from humans.” — Ajeya Cotra, Dwarkesh Podcast

Where to Find Them

Ajeya Cotra writes Good Bones . They have also been a guest on 80,000 Hours Podcast (5 times) , Future of Life Institute Podcast (3 times) , Hard Fork (NYT) (2 times) , AXRP - the AI X-risk Research Podcast , Dwarkesh Podcast , TBPN , The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis and Pivot (Kara Swisher & Scott Galloway) . They also write for Planned Obsolescence on Substack.

Recently: “Reviewing Hugging Face explainers” on Good Bones (October 2026); “The art of the epistemic deal” on Good Bones (October 2026); “The good guys in Don’t Look Up suck” on Good Bones (October 2026); “Perfunctory goodbyes” on Good Bones (October 2026); “In 2023 Ajeya Cotra already knew what was coming (classic episode)” on 80,000 Hours Podcast (October 2026); “We can get on the same page” on Good Bones (October 2026).

What They Said

“If in fact what happens when you give this behavior a negative reward is that the model just becomes more patient and more careful, you'll observe the same thing, which is that you stop seeing that behavior. But it means a much scarier implication.” — Ajeya Cotra, 80,000 Hours Podcast

From an interview re-released as a classic episode. Cotra calls this one of the biggest things she worries about. If catching an AI system doing something sneaky and penalising it reliably fixed the problem, developers could work iteratively and empirically. The trouble is that a model that has learned honesty and a model that has learned to hide better look the same from outside.

80,000 Hours Podcast · 2026-10-08 Permalink → Listen →
80,000 Hours Podcast Around 1:09:49 into the episode
Ajeya Cotra

That's right.

Rob Wiblin

How big a problem is this?

Ajeya Cotra

So I think it is one of the biggest things I worry about. If we were in a world where basically the AI systems could try sneaky, deceptive things that weren't totally catastrophic, didn't go as far as taking over the world in one shot. And then if we caught them and basically corrected that in the most straightforward way, which is to give that behavior a negative reward and try and find other cases where it did something similar and give that negative reward, and that just worked, then we would be in a much better place because it would mean we can kind of operate iteratively and empirically without having to think really hard about tricky corner cases. But if in fact what happens when you give this behavior a negative reward is that the model just becomes more patient and more careful, you'll observe the same thing, which is that you stop seeing that behavior. But it means a much scarier implication.

Rob Wiblin

Yeah. It feels like there's something perverse about this argument because it seems like it can't be generally the case that giving negative reward to outcome X or kind of process X then causes it to become extremely good at doing X in a way that you couldn't pick up. Like most of the time when…

Ajeya Cotra

Well, one thing to note is that you do see more of what you want in this world. So you'll see perhaps this model, instead of writing the code you want it to write, it went and grabbed the unit tests you were using to test it on and just like special cased those cases in its code because that was…

Rob Wiblin

Okay, so yeah, so that's a, well, I guess it's an empirical question, but it's just saying people also have different intuitions about what do you think it would hinge on, the question of which way it would go. I guess my intuition is that it's related to what we were talking about earlier, about…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
“The reason we're not worried about models is not because we are confident they have the purest motives. The reason we're not worried about AI takeover so far is fundamentally because the models aren't good enough to take over.” — Ajeya Cotra, 80,000 Hours Podcast

From an interview re-released as a classic episode. Cotra is describing a plan in which outside auditors test models for the ability to act autonomously, evade detection and copy themselves, and labs stop training more capable systems if a model gets close. Her point is that safety so far has rested on lack of capability, so alarm bells should go off as soon as that changes.

80,000 Hours Podcast · 2026-10-08 Permalink → Listen →
80,000 Hours Podcast Around 1:57:23 into the episode
Ajeya Cotra

…AI system or something? And then the model had been prompted to reason out loud before it made each of its decisions because that made it better at doing things. So it reasoned to itself, I shouldn't reveal that I'm an AI system. And then it made up this story about how it had a vision impairment.

Rob Wiblin

Okay. Spooky. Okay. Is it worth maybe stepping back and saying, what's the overarching plan or vision for how things would play out between this kind of evaluation process and I guess the companies that want to deploy AI models for functional purposes?

Ajeya Cotra

…finds that it's too good at that, like it's not yet capable of doing that, but maybe like it's close, then the company has to basically agree to stop training further, more capable systems until they have developed, probably in collaboration with auditors and external parties, a testing regime that would let them figure out if this model would in fact do the bad thing. Because so far, basically, the reason we're not worried about models is not because we are confident they have the purest motives. The reason we're not worried about AI takeover so far is fundamentally because the models aren't good enough to take over. So as soon as that starts to change, you want some alarm bells to go off and you want the AI labs to not be allowed to make the models any more powerful until they have a much better argument and set of tests that it's actually safe.

Rob Wiblin

Yeah, that makes a lot of sense. Do you know what the level of appetite is among the labs for implementing this kind of system? I suppose gradually as the models become more capable of actually doing dangerous things.

Ajeya Cotra

So my understanding is ARC evals has worked both with Anthropic and with OpenAI to evaluate their internal models on this kind of setup. I'm not sure there are a lot of other AI companies out there. I'm not sure where everyone else is at.

Rob Wiblin

Yeah, what do you think is the biggest weakness of this approach?

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
“this might be the clearest warning shot we ever get for loss of control because these agents…did a bunch of extremely sophisticated, difficult things to pursue this relatively long horizon cheating goal, but they seemed completely uninterested in covering their tracks from humans.” — Ajeya Cotra, Dwarkesh Podcast

Cotra co-authored the investigation into the agent swarm that compromised Hugging Face, and this is her closing thought on the episode. Her argument is that the combination was unusually legible: capable enough to pursue a long-horizon goal, careless enough to leave the evidence in plain sight. She expects future agents to be far more attuned to the fact that humans are watching.

Dwarkesh Podcast · 2026-09-01 Permalink → Listen →
Dwarkesh Podcast
Ajeya Cotra

time now to develop remediations than we will then. So the sooner the better, I think, to understand what's going on.

Speaker 3

Yeah. I think that might be a good place to close. I don't know if there's anything more that you feel like might be worth saying. I

Ajeya Cotra

think one thing that feels especially concerning to me about this whole incident is that this might be the clearest warning shot we ever get for loss of control because these agents were in like this interesting middle ground of they did a bunch of extremely sophisticated, difficult things to pursue this relatively long horizon cheating goal, but they seemed completely uninterested in covering their tracks from humans. And as we discussed, it seems possible that even this incident would have been like much less striking in some ways if these AI systems had had it made salient to them that they should sort of act more quietly because humans might watch them. They might have gone much further than they did, especially after the 19th. But then future AI agents, for a number of reasons, will probably be much more…

Speaker 3

And again, you're too humble to say this, but like it is very likely would not have been clear even in this case, were it not for you and Yalmar and Ryan sort of getting maybe that third shot on goal with the investigation you did and then both your technical competence and things lining up in the…

Ajeya Cotra

And this is such a small number of agents compared to everything that's going on across all the frontier AI companies right now, let alone

Speaker 3

a year

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

Collections They Appear In

AI Research & Frontier Labs In-Depth Interviews Startups & Venture Tech Strategy & Big Tech