Yampolskiy, in a debate with Joscha Bach, defending his estimate that superintelligence almost certainly ends badly for humans. He compares the request for permanent control of a smarter system to asking someone to build a perpetual motion machine: his answer is close to certainty because he thinks the problem has no technical solution.
to continue my answer? Yeah, please have more of a boxing match. I'm not going off. I just need to know what it is.
Apologies. Go ahead, Roman. We'll keep it. We're keeping it very simple. So, P-Doom
is a useful tool. We do risk assessment in all sorts of industries. You know, how dangerous it is to fly, how dangerous it is to drive. The numbers are not randomly generated. I agree that someone with a P-Doom of 46% has a lot of explaining to do us how they got there. Levels around zero and one are much easier. You may have zero concerns because you have a working safety mechanism in place, or you may argue that the problem is impossible. If I ask you about your probabilities of creating a perpetual motion device, I'll say you're going to be around zero, right? The inverse of it, of course, is one. And that's how I feel about our ability to control general superintelligence indefinitely. You can't. It's impossible. You're not going to have a technical solution. What does that mean? It means if you give it enough time, it will probably end up very poorly for us because it doesn't care about you. You're not controlling it. It has other goals and limited resources in our universe and our planet. If it decides to use them for something else, it's not going to care how many of us die in that process. So I think, yeah, PDOM is a useful tool. I don't know if having an extra nine would make me more impactful. Usually, when I meet people who have concerns about AI for different reasons, they worry about loss of meaning, unemployment, discrimination. All those things can be added as increasing overall dangers from this technology. But I think it's not the point. The point is that I'm saying this is equivalent to requesting someone to create a perpetual motion device. You are asking for a perpetual safety device, a system which will never fail once. New models, new data sets, interaction with any agent, malevolent agents, recursive self-improvement, it will never fail once. It's always one step ahead of something smarter than you. I think it's impossible. And so I have a high PDOOM. Is it bigger than others?
Yes. Well, let's go back to 1945, April 1945. The Trinity test. There were a lot of physicists, including some of my late colleagues here, Herb York and others, who were rightfully concerned about some, not the nuclear war aspect of P-Doom of nukes, but of things like igniting the atmosphere, of a runaway, uncontrolled, nuclear, essentially nuclear reaction that would lead to P-Doom of 100%. And there were people like that. I'm not saying Herb York did, by the way, but there were people that believed that. And then later there was the P-Doom of nuclear war. And I want to teleport back. Well, let's say you guys are there. What do we do? Do we not do, do we not create fission, the nuclear fission, because of some very smart people who believed that P-Doom was greater than zero. What would you have done, Roman? Let's go back to that time. So all the examples
with previous technology, whatever it is, nukes or fire, always talk about tools and a human being making a decision. The situation is fundamentally different. We're not talking about a tool. We're talking about agents. They will be independent decision makers. It will not be the standard business as usual where if I understand motivations of this human or I can kill this human, then things will be okay again. This is complete surrender of control. Now, we can talk about at what level of danger you can run an experiment on all the other humans. Eight billion humans, you don't get consent from them. They cannot consent because consent requires understanding. They don't know what's going to happen. So consent is impossible. But we can discuss at what level it's still reasonable to risk their lives for greater good. But that's not even the situation we're in. It's not a single experiment where we're saying if only we allow GPT 6.1 to come out, after that, everything is great. We're talking about all future models. We're talking about recursive self-improvement. We're talking about intelligence explosion. People worry about superintelligence, not in a Donald Trump renaming sense, but smarter than all of us combined. There's a levels to it. System smarter than you, 10x, 100x, million X. Those are not the same problem. To suggest that you will be able to always stay in control of all of them seems insane. But
you couldn't do that with nuclear weapons either. And you've brought up, you know, the bioweapons, and we have to look at today's headlines coming out of Siberia, which is exactly the scenario my friend and guest Annie Jacobson painted in her book, Biological War. And last time you and I spoke, you said, oh, we have treaties about it. Well, Russia is a party of that treaty. And in 1971, when recombinant DNA was invented and created, whatever you want to say, there was no consent either. I guess I want to ask you in a minute what, Roman, take us through the actual steps, because I can think of a million, just like igniting the atmosphere or when the Apollo landers were going, they thought they would sink to the center of the moon because they didn't know about the structural integrity of the regolith on the moon. You know, there's millions of scenarios you can't do counterfactuals on. And I want to ask you, let's take me step by step through actual superintelligence leads to doom. I want to ask you that in a minute. But Yoshi, I want to give you a chance to respond to what Roman's saying, because it is an argument that I have a lot of respect for. I don't agree with Roman, but I have a lot of respect for him in that we can't give consent. We haven't given consent despite all these things. So do we need to have consent before we shut things down? I mean, at what point are you comfortable with putting the Gini back in the bottle if Roman's correct? I mean, your P doom might be irrelevant, but how do you speak for the rest of humanity? Not you personally, but how do we as scientists or as technologists?
