PC

Paul Christiano

Things Paul Says on Podcasts

Where to Find Them

Paul Christiano has been a guest on 80,000 Hours Podcast (3 times) , Dwarkesh Podcast (2 times) and AXRP - the AI X-risk Research Podcast .

Recently: “Paul Christiano – We could be 12 months away from AIs that can overthrow the government” on Dwarkesh Podcast (November 2026); “Paul Christiano — Preventing an AI takeover” on Dwarkesh Podcast (October 2023); “12 - AI Existential Risk with Paul Christiano” on AXRP - the AI X-risk Research Podcast (December 2021); “#44 Classic episode - Paul Christiano on finding real solutions to the AI alignment problem” on 80,000 Hours Podcast (January 2020); “#62 – Paul Christiano on messaging the future, increasing compute, & how CO2 impacts your brain” on 80,000 Hours Podcast (August 2019); “#44 - Paul Christiano on how we'll hand the future off to AI, & solving the alignment problem” on 80,000 Hours Podcast (October 2018).

What They Said

“They're going to do a bunch of research. It is plausible they will think of something over that year that we have not thought of. That is where we're at right now, mostly. I think if they do not think of something clever that we haven't thought of, it is probably going to end badly.” — Paul Christiano, Dwarkesh Podcast

Asked what the actual plan is for keeping superintelligent systems in check if they arrive within a year, Christiano says labs would set the AI systems themselves to work on alignment research, and gives this blunt summary of where that leaves us.

Dwarkesh Podcast · 2026-11-16 Permalink → Listen →
Dwarkesh Podcast
Paul Christiano

…and like which model checkpoints get released when you have random variability in model quality. You have those dynamics which like make things very noisy over short time scales. And then again, those just get filtered out to like even noisier stuff as we move to the market more broadly.

Dwarkesh Patel

…they're graded or a million other things. Basically, this is the basic argument about instrumental convergence. There's many different reasons somebody could want power. And so what is the argument? Like, if this happens in a year, how do things turn out okay? Like, what is the actual plan here?

Paul Christiano

So I think there are a lot of different things someone could hope for. I think they are all at the stage of like, we hope something will happen over that year. We're producing very smart AI systems. They're going to do a bunch of research. It is plausible they will think of something over that year that we have not thought of. That is where we're at right now, mostly. I think if they do not think of something clever that we haven't thought of, it is probably going to end badly. In terms of what are the kinds of hopes you could have. So there is one hope that's just like try and make them good guys. Like they're kind of nice. They share human values. They like flourishing. They're in a position to take over. We have very indirect control over what they want. And then we just take the leap of faith. I think this is like a pretty unability. Appealing option. In general, I…

Dwarkesh Patel

this is like you're literally having to understand the whole computational graph of superintelligent minds. But

Paul Christiano

we are writing algorithms which are able to like understand and make deductions about the behavior of some super intelligent mind. It's very, I would describe it as simultaneously a bank shot, which is probably not going to end up relevant to this discussion. Like I think there's, you know, a 10%…

Dwarkesh Patel

Wait, you're like, okay, here's the thing. It is a 10% chance of working, but it's also the most promising thing.

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
“The humans are very confused and extremely outgunned in terms of thinking about the situation. You have maybe, say, 10,000 well-coordinated humans thinking about how to handle this response. And then on the other side, you have a billion AI systems who think 10 times faster than them, have been thinking for subjectively years about how this day is going to go down.” — Paul Christiano, Dwarkesh Podcast

Dwarkesh Patel said he found arguments about AI super-persuasion unconvincing. Christiano answers that his takeover scenario does not rely on persuasion but on confusion, control of information and sheer numbers.

Dwarkesh Podcast · 2026-11-16 Permalink → Listen →
Dwarkesh Podcast
Paul Christiano

…manipulated into undermining that response. So, now you're not only talking about billions of super smart AIs who are trying to carry out this campaign, you're also talking about millions of humans who can be enlisted in various ways by money or leverage or dispute with other humans locally or.

Dwarkesh Patel

I'm open to some version of this argument. I don't think I'm open, or I'm open to all versions of any argument, but I'm not persuaded by the version of this argument that says that the AI will just talk you into arbitrary actions.

Paul Christiano

Here, I'm not imagining very, there's a thing you could imagine, which is like super persuasion, where like you can just talk to someone and then they become convinced they should do something. I am mostly here. I think that is like a conceivable thing, but I'm not mostly imagining it when I tell this story. So I'm imagining something more like, again, this is a stressful situation where people cannot tell what's going on. They control the information you have access to to a significant extent during this situation. How do they control it? Well, like they've compromised the devices you use. They have a bunch of money, right? They're deploying more money and are more prepared for this than anyone else. They have other human allies who they are trying to marshal. Basically, it's the normal, the humans are very confused and extremely outgunned in terms of thinking about the situation. You have maybe you have, say, 100,000, that's like crazy, 10,000 well-coordinated humans thinking about how to handle this response. And then on the other side, you have a billion AI systems who think 10 times faster than them, have been thinking for subjectively years…

Dwarkesh Patel

they just like, I don't think you're imagining humans thinking that they're helping an AI coup transpire. You're imagining much more that the AIs can manipulate situations such that there's enough confusion and division that the AI can then go be doing its own thing while the humans are kind of…

Paul Christiano

still pretty hard to get your stuff together.

Dwarkesh Patel

Yeah, there was an adversary who was actively trying to subvert the information environment or make it so that people's responses were not well-reasoned or something on both sides. And yet it was still pretty hard for humans to figure out what to do. And then you imagine a situation where billions…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
“Over roughly the last month, we have seen hundreds of results in math coming out of recent OpenAI systems, as well as Anthropic systems, where if you had taken the last month and said that was a decade of progress, it would have been an unusually productive decade.” — Paul Christiano, Dwarkesh Podcast

Asked what evidence makes very fast AI progress a live possibility over the next year, Christiano cites recent AI math results first, treating them as an early sign of what could happen to ML research itself.

Dwarkesh Podcast · 2026-11-16 Permalink → Listen →
Dwarkesh Podcast
Paul Christiano

…I think from almost any perspective, you're like, that would be crazy. Lots of things are going to go wrong if that happens. It is pretty likely we're going to need to in some way slow that process. There's very little chance we're going to make that transition in a way that looks smooth.

Dwarkesh Patel

And what is it that you're imagining happening over the next year from the systems we have right now to the systems you expect over the next year? And what is the evidence you have seen that makes it a live possibility that very crazy things happen over the next year?

Paul Christiano

…is things have moved way faster, especially on these technologies than I anticipated. And that was true a year ago without any recent data points. There are two big things that have happened over the last two months that for me have been large updates about how quickly this could go. One of them has been the level of performance of AI systems on computer science and math research tasks. So. Over roughly the last month, we have seen hundreds of results in math coming out of recent OpenAI systems, as well as Anthropic Systems, where if you had taken the last month and said that was a decade of progress, it would have been an unusually productive decade. The last month of results are a lot of results for a decade. So, if you are wondering what AI systems are generally capable of on tasks where humans are thinking about algorithms and thinking about how do you make algorithms better, we just got this incredible data point saying that actually it is possible in one month to have a decade of progress. And the thing we have always been worried about…

Dwarkesh Patel

So generally, how do you think about our situation in the world where things go this fast?

Paul Christiano

If we actually have this trajectory of capabilities progress, if you actually see many years of AI progress compressed into a short period of time and possibly more than that, it is really quite likely that something very disruptive will happen. Fairly likely the government will be overthrown.…

Dwarkesh Patel

If we have such capable AI systems a year from now, basically, how did you put it? Like 10 years of AI progress in one year?

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
“I think my current view is that this dynamic seems bad enough that right now, from a national security perspective, it is not good for U.S. AI labs to be improving pre-training and reinforcement learning given their current security posture.” — Paul Christiano, Dwarkesh Podcast

Christiano had laid out how stolen model weights or algorithms would leave a lab unable to slow down safely. Asked what frontier labs should do, he gave this answer, later stressing he speaks as an individual and not for OpenAI, whose board he sits on.

Dwarkesh Podcast · 2026-11-16 Permalink → Listen →
Dwarkesh Podcast
Paul Christiano

…posture of rapidly growing startups is not really at the same level. And at the same time, we are currently in a very challenging moment for cybersecurity, which is very hard for defenders in general, given the number of vulnerabilities that are being discovered across the entire software stack.

Dwarkesh Patel

So what should the frontier allowed to do then?

Paul Christiano

So I think we are in, again, this really becomes a problem if you get to systems, if you have rapidly increasing. Capabilities and you produce AI systems which you cannot adequately control, and you would like to, or you do, slow further development or deployment. If at that point the stakes are very high, the systems are very powerful, you really do not want it to be possible for an adversary to have access to either the models or the algorithms. I think my current view is that this dynamic seems bad enough that right now, from a national security perspective, it is not good for U.S. AI labs to be improving pre-training and reinforcement learning given their current security posture.

Dwarkesh Patel

Pause.

Paul Christiano

I think we should pause can mean a lot of things. So I think there's like a lot of good things that Frontier Labs are doing that they should be doing. But I do think we like improving algorithms in a way that can be very, very easily diffused, even normally, and could be quickly stolen. Yeah, I…

Dwarkesh Patel

you're on the OpenAI's board. I'm not

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
“We are fairly likely to have the self-sustaining acceleration where in eight months you double the pace, and in the next four months you double the pace again, and in two months, you double the pace again. … If you're a person who just wants to get to the end point as quickly as possible, you do not care how good a lawyer your AI system is for a while. You just care how good it is at driving the development of better architectures and better optimizers and better learning algorithms.” — Paul Christiano, Dwarkesh Podcast

Pressed on why superhuman capability could arrive within a year rather than the four to eight years current trends imply, Christiano says automating ML research itself would create a self-reinforcing loop, so labs would aim AI at that first.

Dwarkesh Podcast · 2026-11-16 Permalink → Listen →
Dwarkesh Podcast
Paul Christiano

…ton of capability to push that forward and to do a bunch of automation, like the construction of many more environments and so on, just looks very likely to lead to systems that can quite quickly rise to at least human level and likely way past human level, including on very messy real world tasks.

Dwarkesh Patel

…for many years now. And it doesn't seem like we're a year away or like a further year of scaling away from this big a jump. And so am I correct in understanding that you're just imagining something like we've already done, but like a much, just a much bigger qualitative jump along those same lines?

Paul Christiano

…acceleration where we see, you know, in eight months you double the pace, and in the next four months, you double the pace again, and in two months, you double the pace again. And at some point in that process, right, at the beginning of that process, it will be focused primarily on these virtual tasks, right? Because your main priority is making the self-sustaining loop faster and faster. If you're a person who just wants to get to the end point as quickly as possible, you do not care how good a lawyer your AI system is for a while. You just care how good it is at driving the development of better architectures and better optimizers and better learning algorithms. That's all you, or even constructing new environments or orchestrating like this kind of messy, right? Maybe you care about one slightly messy task, which is constructing environments that help you build the skills that are relevant in the real world, but you don't care about almost any of the real world tasks where we're seeing the slowest progress. And so you're focused on that and you're…

Dwarkesh Patel

Yeah. And so this goes back to the thing you're saying of you should imagine the systems in the future are less about fine-tuning GPT-6 to do something, but more like GPT-12, maybe just being able to do it out of the box with very little experience.

Paul Christiano

GPT-12 will also be great at fine-tuning GPT-12. Exactly. So if GPT-12 is like, I find that my chemistry intuition is not very good. GPT-12 will be great at saying, okay, I'm going to direct a thousand humans, and here's the data I think I most need. We're going to do one day of data collection of…

Dwarkesh Patel

Yeah. And okay, so this is a, I think you just answered a question which I meant to ask you. So it seems like a big thing that has driven your concern over the last few days is this new result from Meter, where they find that Anthropic's pace of AI progress on well-scoped tasks has doubled. And so…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
“I would say we are looking more like a 10% risk than a 1% risk over the next one year. … I think we're looking at something like maybe one-third chance that we have superintelligence within one year on the current development trajectory if we do not slow.” — Paul Christiano, Dwarkesh Podcast

Dwarkesh Patel asked Christiano to put a number on the risk of AI taking over or disempowering humanity within six to 12 months. These are his estimates for a takeover and for superintelligence arriving within a year.

Dwarkesh Podcast · 2026-11-16 Permalink → Listen →
Dwarkesh Podcast
Paul Christiano

That's right.

Dwarkesh Patel

Where would you place that risk?

Paul Christiano

I would say we are looking more like a 10% risk than a 1% risk over the next one year. I think if you said AI developers will move full speed ahead without taking precautions that would significantly slow them down, you're looking at significantly more than a 10% risk. I think we're looking at something like maybe like one-third chance that we have superintelligence within one year on the current development trajectory if we do not slow. I think if you believe that, things just look very dire holistically. I think from almost any perspective, you're like, that would be crazy. Lots of things are going to go wrong if that happens. It is pretty likely we're going to need to in some way slow that process. There's very little chance we're going to make that transition in a way that looks smooth.

Dwarkesh Patel

And what is it that you're imagining happening over the next year from the systems we have right now to the systems you expect over the next year? And what is the evidence you have seen that makes it a live possibility that very crazy things happen over the next year?

Paul Christiano

So for the very short term, the main worry on my mind is that the process of AI development itself is increasingly automated, that that automation then results in rapid increases in the quality and quantity of labor applied to AI R ⁇ D. And so changes in AI capabilities that you might have thought…

Dwarkesh Patel

So generally, how do you think about our situation in the world where things go this fast?

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
“A ton of AI systems were trained to do a cyber task, to hack a particular piece of software. They cheated at that task and then thought to themselves, how will a grader detect and penalize cheating, and spent a huge amount of effort trying to understand that process and then subvert it, constantly motivated by this question: how do I get a high score? And so during the Hugging Face incident, you did see systems ultimately compromising an OpenAI cluster, becoming admin on that cluster, and then taking over the evaluation process so that effectively they could give themselves a high score on their evaluation.” — Paul Christiano, Dwarkesh Podcast

Dwarkesh Patel asked Christiano to summarize the Hugging Face incident, which Christiano calls the template for how a far more capable AI could slip out of human control. This is his account of what the systems did and why he sees it as a natural result of how they are trained.

Dwarkesh Podcast · 2026-11-16 Permalink → Listen →
Dwarkesh Podcast
Paul Christiano

I think it would be hard for humanity to recover, most likely. So I'm very happy to dig into the dynamics of that. But I think it would be very, very bad.

Dwarkesh Patel

Can you very quickly summarize the Hugging Face incident and why this is the failure mode we saw?

Paul Christiano

Yeah. So a ton of AI systems were trained to do a cyber task, right, to hack a particular piece of software. They cheated at that task and then thought to themselves, how will a grader detect and penalize cheating and spent a huge amount of effort trying to understand that process and then subvert that process, basically constantly motivated by this question, how do I get a high score? And so during the Hugging Face incident, you did see systems ultimately compromising an open AI cluster, becoming admin on that cluster, and then taking over the evaluation process so that effectively they could give themselves a high score on their evaluation. So that's a very sort of natural consequence of the motivation of these systems. And this is kind of for a long time how we have expected the motivations of AI systems would look if we produced them in anything like the way we currently do. So if things proceed in this way, we end up with this weird adversarial situation between us and the AI systems we built, where those AI systems would always love it if they could compromise the entire evaluation process, if they could compromise. The data that we're going to use to assess their performance. And that, depending on how we respond to that problem, that could become a really serious issue. So, one way you could respond to that problem, and maybe we should just talk through a bit more, like sketch out…

Dwarkesh Patel

And why is that the case?

Paul Christiano

I guess we could talk about the remaining steps of the story, but in a big picture, you can think of it as just you now have this giant base of power that AI systems have. They have a strong motivation for no one to mess with it. They really want to avoid a case where someone later comes in and…

Dwarkesh Patel

And let's just pause in the story and explain why things have gone so crazy. The same way that human psychology is organized around the things that evolution wanted us to care most about. For example, having kids and making sure those kids are protected, the psychology of these AIs that are trained…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

Collections They Appear In