Doom Debates! · AI Research & Frontier Labs · November 2026
His guest had argued that AI will carry human values because it was trained on human data. Shapira replied that a system far up the intelligence scale would stop resembling its origins. It is the core of their disagreement.
…right? They boost themselves up to being super intelligent in like this path-dependent way, right? So they end up as these different types of super intelligence and then baked into the type of intelligence they are, their preferences will be baked into that. Is that kind of your argument?
…of how humans are intelligent. But the reality is that we produce these artifacts, this data, which is a reflection of our intelligence. And that has been used as the training for these AI. So it's not at all surprising that they are going to pick up on many of the same patterns that humans do.
I think you're missing something here. It's this idea of the intelligence scale. By the time you get really high on the intelligence scale, by the time you have the essence of intelligence to a high degree, the bootstrapping that you did to get there, you no longer see the signature of your ancestor as like an intelligent ape. Like I don't expect that signature to show up very much. So just some analogies. Imagine that you made a submarine and the submarine could swim about as fast or as skillfully as a fish. In that case, if you say, okay, the submarine kind of looks like a fish, it has like flippers, I'd be like, okay, yeah, that's possible. But if you told me that the submarine is a thousand times, it carries, it goes a thousand times faster than a fish and it weighs a million times more than a fish. At that point, I don't expect it to have much of the fish signature in it. I expect it to be shaped by the requirements of the field of swimming. So similarly with intelligence, maybe I have to, there's like a lemma, there's a sub-belief here. Like I have to make sure that you believe that there is such a thing as being a thousand times as smart as a…
Well, I think you have to look at how the intelligence was developed, right? So kind of like how the submarine was developed or whatever. And in this case, at the current state of technology, it seems pretty clear to me that the learning mechanisms, the neural network learning mechanisms, deep…
I hear you, but ants, I mean, give me a break. Like we're talking about super intelligence here, okay? I just, when you have a super intelligence, okay, what's a what's a useful mental model for a super intelligence? In my mind, it looks like something that models a desired outcome, an outcome that…
So, I mean, it's maybe helpful to have a way of understanding any kind of intelligence. And so the way I think of it is that all intelligences can be thought of kind of as having input, doing some processing, and then creating output at the most basic level. And you can apply that at the level of a…