“It's sort of like if someone were to be impressed that I woke up this morning and I prophesized, I will soon have a cup of coffee. And then amazingly, I got up, I made a cup of coffee and I drank it. So it was like, wow, that's incredible that you accurately predicted what was going to happen.”
Deep Questions with Cal Newport · October 2026
Cal Newport raises a common defence of the early AI-risk thinkers: they may be odd, but they were right. Torres answers that the leading AI companies grew directly out of that same movement, so its members were predicting things they went on to build. Torres calls it a self-fulfilling prophecy rather than foresight.
Transcript
Right. So it's exactly the kind of profoundly pernicious reasoning that Hagstrom highlights in the passage, right? So Yakowski is saying, like, okay, there's a thermonuclear war. Like, probably it's probably not the case that everyone will die. And the best calculations that I've seen from the past six, seven years support that. Like, like maybe 5 billion people would die, but that would leave a reassuring 3 billion to rebuild civilization. So he's saying that we should risk thermonuclear war. If that is what is necessary to prevent misaligned superintelligence, then we should do that. He was asked online, quote, how many people are allowed to die to prevent AGI, end quote, from being built in the near future, because it would be misaligned, he says. And his response is like, basically, everyone, as long as there are enough survivors in close proximity to carry on the human species to rebuild civilization, and that I'm quoting, then there's still a chance of reaching the stars someday. Well, if you like, just consider like what that means. The minimum viable human population is estimated to be between 40,000 and maybe as low as 150 people. So pretty, like nobody knows. But even if it's just 40,000, if you subtract 40,000 from 8.3 billion, you get just rounding off 8.3 billion. So he's saying 8.3 billion people should be, quote, allowed to die to ensure that superintelligence is aligned. Again, he is pro-superintelligence. If we figured out a way to align superintelligence to the Tescural worldview tomorrow, he would come out tomorrow and say, pedal to the metal. Build ASI as quickly as possible. We'll create post-humanity. It'll be unethical to have biological humans. We'll wipe them off Earth. We'll replace them and so on. And so it's just important to understand how absolutely extreme that view is. But you're completely right. That utopia is just, I think, a very dangerous concept. And the other thing I would add to this is I have been speaking so far as if the successful realization of the Tescral vision is possible. And I've been saying that if they successfully realize their vision, then the outcome of safe superintelligence will become completely irrelevant. Or according to many of them, like Peter Thiel, we should disappear. I actually am very skeptical that they could successfully realize this vision. But you brought up utopian movements of the 20th century. And examples go back many centuries before that. These utopian movements never reached their goal. But that doesn't mean that they weren't profoundly dangerous. They left a trail. In so many cases, they left a trail of destruction behind them. I mean, Hitler promised a thousand-year Reich. He basically was borrowing from Christian eschatological, the Christian eschatological narrative, which promises a millennial kingdom at some point in the future. One of the bloodiest battles in human history was the Taping Rebellion of the 19th century in China and resulted, I don't know, like something like 30 million deaths. I can't remember the exact figure, but this was driven by the Taping Heavenly Kingdom, which was essentially a utopian apocalyptic movement, just like Tescril. Terrorist groups, doomsday cults, the Islamic State, they were an apocalyptic movement. Well, from the religious point of view, the other side of the apocalypse is utopia. So it's just another way of saying that they were motivated by utopian visions of the future. And that's why they were trying to, literally, they were trying to start the Islamic version of the Battle of Armageddon in this small Syrian town in northern Syria near Aleppo called the Tabik, which is said to be the location of their version of the grand battle. So it's just my main concern is that so far, the AI Doomer have left a trail of destruction behind them. I mean, some people would argue that they are tearing apart society. They're flooding the internet with disinformation, deep fakes. They're working with the US military to engage in illegal activities like kidnapping Maduro and bombing Iran. They're partnering with Palantir to sort of like build the infrastructure for a totalitarian state. And so all of these are, I mean, not to mention the massive, massive intellectual property theft that they are all guilty of. So these are like harms that we can point to right now. And I think the harms are going to get worse. But from their perspective, they're marching to utopia. And sometimes the march of progress, it's going to trample some people. And also, from the grand cosmic point of view, all of these harms that they're causing right now, they are mere peccadillos, right? They're just small, like little mishaps that utopia will forgive because utopia forgives everything.
So, this brings me to the last thing I want to ask you about before I let you go is trying to understand, okay, what's happening right now. And I will say, I'm sure you've experienced this as well. But I find if I write or talk about some of these movements, a very common response I get is like, you're just being ad hominem. Sure, these are weird people, but they were right. The things they were warning us about are starting to happen. So who cares if they're weird or not? They were right. Now, I have this alternate view I want to run by you that actually it's a self-fulfilling prophecy, right? So that they're talking about AI in the ways that matches the Tescrell sort of predictions because they are from Tesgreal and they're trying to bring those predictions about. And I also am increasingly believing, if you look at Anthropic OpenAI, is that they're specifically running experiments and developing this technology in a way that's creating dangers, like you were just talking about, the dangerous experiments, haphazard experiments, unpredictable experiments, exactly because they're trying to rush forward a movement towards their predicted sort of superintelligent future. The strong version of this argument would say if we go to a counterfactual world where you don't have these two Tescriel-aligned labs at the frontier and you just have other technology companies unrelated to these philosophies pushing for generative AI, we would be having a completely different conversation. There would be no conversation about existential death, the sort of really unpredictable, scary experiments like what happened at OpenAI the summer just would not have happened. No one responsible would have run those. And that so that we have almost, we're already seeing this damage on the Ralph Utopia. It's not that they predicted this right. It's that they are trying to make their predictions happen. But I want to get your final take on this is how do we understand what is happening right now, the things that are concerning people in the AI world, how that connects to the Tescriel philosophy.
Yeah, I think you are exactly right. So one way I put it in the past is if you want to understand, this kind of brings us full circle, if you want to understand the behavior of companies, virtually all companies in our capitalist milieu, you have to look no further than the profit motive, right? Why are the fossil fuel companies destroying the world for profit? Why are pharmaceutical companies jacking up prices? Well, they just want to increase the revenue and so on and so on. If you look at the AI Doomer, they are different. They are anomalous. And I can't think of any other examples, right? I said before that profit has become an increasingly significant factor that explains kind of their behavior, the dynamics, the race dynamics they're in. But if you look at the origins and if you look at the core of these companies, what is really ultimately driving them, it is ideological. It's Tescrill. Tescruel is integral to an explanation of why there's an ASI race, why these companies arose in the first place. All of this is to say that because these companies directly emerged out of the Tescrill movement, claiming that Teskruel advocates back in the day were prescient for talking about AI risks and so on is a complete misreading of the situation. It's sort of like if someone were to be impressed that, you know, I woke up this morning and I prophesized, I will soon have a cup of coffee. And then amazingly, I got up, I made a cup of coffee and I drank it. So it was like, wow, that's incredible that you accurately predicted what was going to happen. Well, yeah, because like, you know, I mean, DeepMind got its funding because Debis Hasabis and Shane Lake, two of the three co-founders, gave talks at the Singularity Summit. Singularity Summit was founded by Jakowski, Ray Kurzweil, and Peter Thiel. Jakowski was funded by Peter Thiel. The connections here are absolutely convoluted and completely incestuous. I've written an article about this, but it's stunning when you, I mean, it looks like a conspiracy when you actually look at the details. And, you know, not all conspiracies are conspiracy theories, but it's just like an incestuous network of these people like exchanging money and supporting each other. So they founded the Singularity Summit. Debis Hasabis and Shane Lake were invited. They gave a talk about superintelligence because Shane Lake had just published his dissertation on superintelligence, for which he won a prize from Jakowski's Singularity Institute for $10,000. And then after the Singularity, Summit event, they followed Peter Thiel back to his mansion. And Peter Thiel is as test real as anyone else in this community. They asked him for funding and he gave them funding, right? So, I mean, DeepMind, which was the company that, in a sense, you could say kind of initiated the super intelligence race because OpenAI was founded because there were concerns about Demis Hosebus establishing what they refer to as an AGI dictatorship, right? Basically, just take over the world. And so the whole reason that DeepMind came into existence is because Shane Lake was deeply involved in this community. Demis Hosebus was into the idea of the singularity, and they gave talks at this conference co-founded by Yitkowski. So to say that Yakowski was clairvoyant in some sense for having anticipated what's going on now is just to completely misinterpret the situation. You're totally right that this is a self-fulfilling prophecy. And so we have them to blame for the current predicament. Nobody has done more than the Tescural people, including the Doomers. Nobody has done more than the Doomers to launch, sustain, and accelerate the race to build superintelligence. So rather than saying they're prescient, you should point your finger and blame them for putting us in this situation. Because my personal view is that if there had not been a test grill movement, there would not be a race to build these general systems that exceed human capabilities in every cognitive domain of interest. I don't think that would exist right now. So they are responsible for the unfortunate plight that we all now find each other, ourselves in.
Well, I think, Emile Torres is vitally important information. It's confusing. So I'm glad we got to spend some extra time to try to pull this apart. And I've been saying this to my audience a lot. You have to understand this. This is not ad hominem. This is not, well, they might be right, but they're weird. It's the actions we are seeing by certain companies, the rhetoric we're seeing by certain companies, as you said, is in an almost unprecedented situation. I also tried to come up with an example from American history at least, where you had a really large corporation that was not being driven by the profit motive, but instead by a sort of philosophical, sort of religious motive. You have to find, I guess, a sort of explicitly religiously aligned organization, I guess. So it's really important to understand what's going on. And you've seen it from both sides. So I appreciate you coming on. I want to point people towards all the excellent work you do on this. Your website is, it's X Riskology. Am I getting that right?
Yes, but people should really go to my newsletter.
Newsletter. Okay, go for it. Yeah.
More from Deep Questions with Cal Newport
