RW Robert Wright Robert Wright Author and podcaster who writes about evolution, psychology, religion and the prospects for global cooperation, now with a focus on AI and how to avert catastro...

“You could say the Boeing Corporation is a superintelligence. No one human could know how to build an airliner, but the Boeing Corporation is an information processing system that collectively knows that. And from a sufficient distance, it just looks like a giant brain.”

Doom Debates! · AI Research & Frontier Labs · October 2026

“You could say the Boeing Corporation is a superintelligence. No one human could know how to build an airliner, but the Boeing Corporation is an information processing system that collectively knows that. And from a sufficient distance, it just looks like a giant brain.” — Robert Wright, Doom Debates!

Wright was explaining why even well-informed people have trouble picturing superintelligence. His answer is that it may not be one very smart model but many pretty smart ones working together, the way a company knows things none of its employees do. He goes on to apply the same idea to swarms of AI agents.

Doom Debates! · 2026-10-02 Listen to the episode → More from Robert Wright → More AI Research & Frontier Labs quotes →

Transcript

Doom Debates! Around 51:44 into the episode
Liron Shapira

been compared in the past when I was even nerdier than I am now. I've been compared in the past to Sheldon from, you know, the Big Bang Theory.

Speaker 2

Okay, that's a that's a that

Robert Wright

that that that's it. My generation is the Poindexter generation. That's our icon, and yours is Sheldon. The um, so yeah, I mean, there's another thing going on here, uh, in terms of people having trouble, maybe still, including, I, you know, I'd say smart, well-informed people, imagining pretty well-informed people, imagining what's coming. And it was so well illustrated by the Open AI breakout. It's like, you know, when you ask what is super intelligence, I've said this before, but one way to say it is like, you know, you could say the Boeing Corporation is a superintelligence. No one human could know how to build an airliner, but the Boeing Corporation is an information processing system that collectively knows that. And from a sufficient distance, it just looks like a giant brain. And when you put a lot of pretty smart brains together, like the people who work at Boeing, you get pretty amazing shit that no single brain could do. And it's the same way with agent swarms, you know? And, you know, as I've said, it's Like, well, what's the difference between AGI and superintelligence? Well, maybe just a difference between an LLM and the agent swarm it can sponsor, right? I mean, and these things, you know, they communicate at lightning speed and think at lightning speed. And we're just seeing the beginnings of it, okay? This was a thousand. There's no law of the universe that says you can't have 10,000 agents that are based on the next generation of LLMs.

Liron Shapira

Yeah, so let's help the people with their ontology, right? Because the same way that we would have liked people to realize that the chatbots aren't going to look like chatbots, right? So today they look like agents, but the agents still stay in your computer and the agents still have an off-button, right? I think people still imagine the off-buttons. I think one thing you and I can probably warn people is like, they're probably going to defend themselves from being turned off or make it hard to be turned off soon, right? That's one of the properties we can probably expect.

Robert Wright

Yeah, I think right now, you know, it's great that the discourse has moved to a point where people are demanding some kind of governmental action. But it's true that you're hearing things like build an off switch and make super intelligence illegal. I'm not saying those are bad things. It might be progress to try to articulate the legislation and pass it. That might help. But I do want to say that imagining a kind of off switch that's going to handle the challenges that you and I think are coming with the next generation or in a way exist now is not trivially easy. And like defining super intelligence, if you want to ban it, is challenging. Plus there's a problem that, as I just said, if you look at an LLM and say, well, this isn't super intelligence, so it's okay. But then look at what 10,000 agents coming out of it look like, you may be at super intelligence, right? So those, you know, we need something firmer. I think we need to be thinking very seriously about at least a pause in big training runs. I mean, these companies have models we haven't even seen already. Okay. Those training runs have happened. And we've got plenty of intelligence.

Liron Shapira

Yeah, I want to help people understand that it's, you know, the type of thing it is, right? It's not chatbot and it's not exactly even agent. I mean, in a technical sense, it still is because agent just means outcome steering entity. So it is still going to be a lot of fun. It has a goal,

Speaker names from our own diarization · position estimated from where the line sits in the episode

More from Doom Debates!

“I believe in the Icarus curve. So we're Icarus right now and we're going up because we're flying closer and closer to the sun. And that's great. Flying closer to the sun is better, right? You get a better view. It's fun. But eventually your wings burn off and you just plunge into hell and it's game over.” — Liron Shapira, Doom Debates! “A chatbot is like somebody you visit in the hospital who can't get out of bed. They can talk to you, they can draw pictures, but they're not going to get up and change the thermostat, much less stage a coup and take over the hospital. But once you're talking about agents, you're just in a whole different world.” — Robert Wright, Doom Debates!