Yampolskiy, in a debate with Joscha Bach, defending his estimate that superintelligence almost certainly ends badly for humans. He compares the request for permanent control of a smarter system to asking someone to build a perpetual motion machine: his answer is close to certainty because he thinks the problem has no technical solution.
to continue my answer? Yeah, please have more of a boxing match. I'm not going off. I just need to know what it is.
Apologies. Go ahead, Roman. We'll keep it. We're keeping it very simple. So, P-Doom
…nine would make me more impactful. Usually, when I meet people who have concerns about AI for different reasons, they worry about loss of meaning, unemployment, discrimination. All those things can be added as increasing overall dangers from this technology. But I think it's not the point. The point is that I'm saying this is equivalent to requesting someone to create a perpetual motion device. You are asking for a perpetual safety device, a system which will never fail once. New models, new data sets, interaction with any agent, malevolent agents, recursive self-improvement, it will never fail once. It's always one step ahead of something smarter than you. I think it's impossible. And so I have a high PDOOM. Is it bigger than others?
Yes. Well, let's go back to 1945, April 1945. The Trinity test. There were a lot of physicists, including some of my late colleagues here, Herb York and others, who were rightfully concerned about some, not the nuclear war aspect of P-Doom of nukes, but of things like igniting the atmosphere, of a…
with previous technology, whatever it is, nukes or fire, always talk about tools and a human being making a decision. The situation is fundamentally different. We're not talking about a tool. We're talking about agents. They will be independent decision makers. It will not be the standard business…
you couldn't do that with nuclear weapons either. And you've brought up, you know, the bioweapons, and we have to look at today's headlines coming out of Siberia, which is exactly the scenario my friend and guest Annie Jacobson painted in her book, Biological War. And last time you and I spoke, you…
