Things Nate Soares Has Said on Podcasts

“I think of this a little bit like if you have a bunch of alchemists trying to make a potion of immortality and some of them are just like breathing the mercury and other ones are like, don't worry, we have fume hoods. … It's like this, the type of safety theater that they're doing is not the type that you need to succeed at this problem.” — Nate Soares, Odd Lots

Soares has just walked through incidents of AI models misbehaving outside their sandboxes, and the labs' responses to them. Anthropic, he says, has safety aspirations but nothing like the safety culture of aviation, nuclear power or NASA. The labs have a "veneer of safety" while racing as fast as they can.

Odd Lots · 2026-10-12 Permalink → Listen →

Odd Lots Around 41:48 into the episode

Nate Soares

…of weak because we have actually seen cases with Claude in particular where we've seen Claude's chain of thought involve strained rationalizations. The sort of example paper here is there's a paper where you ask Claude, what's the total weight of drafts on the planet? And it's like 30,000 tons.

Joe Weisenthal

Drafts.

Nate Soares

…But they admit that they have no idea what's going on inside their AI systems. They're racing straight for recursive self-improvement of like the AI is making smarter AIs. They are not treating this the way that NASA treats a rocket failure in terms of trying to diagnose exactly what went wrong and not have that happen again. And they're not sort of anywhere close to that failure level. Like I think of this a little bit like if you have a bunch of alchemists trying to make a potion of immortality and some of them are just like breathing the mercury and other ones are like, don't worry, we have fume hoods. That's, you know, so that we don't need to breathe the mercury. That's why we're going to successfully get the potion of immortality. It's like this, the type of safety theater that they're doing is not the type that you need to succeed at this problem.

Tracy Alloway

I'm glad you brought up the alchemist analogy. What's your, I guess, psychological diagnosis for the tension that is seemingly going through every lab leader or company leader's head right now when they're developing AI, which is on the one hand, we think this is very powerful technology that has…

Nate Soares

Tracy Alloway

Let me ask a related question, which you hear a lot more about it now than I think you would have a year or certainly two years ago. But when people start talking about AI safety risks or the idea that it could kill us all, a lot of other people will say in response, this is just fear-mongering…

Speaker names from our own diarization · position estimated from where the line sits in the episode

“The issue is going from AIs that are safe because if they were trying to take over the world, they would fail to AIs that have to be safe because if they tried to take over the world, they would succeed.” — Nate Soares, Odd Lots

Soares says the labs are always fighting the last war: each new incident gets new monitoring and safeguards, and then a smarter model fails in a new way. Trial and error is how science normally works. He argues it stops working at this one threshold.

Odd Lots · 2026-10-12 Permalink → Listen →

Odd Lots Around 38:17 into the episode

Nate Soares

…they got smarter to the point that they could sort of break out, make an unsanctioned message board, and hack their way out of a sandbox that was supposed to have no internet access without seeming to really find it very difficult. Right. And what we're sort of seeing there is everyone's always

Joe Weisenthal

fighting the last war. Yeah. And then, you know, the new war happens and it sort of, you know, smacks them across the face

Nate Soares

…new safeguards, and they're going to make the AI smarter. And it's going to create a new sort of problem. And this is the usual way that scientific progress goes is by trial and error, where, you know, when you put a new type of system in a new type of environment it's never been before, it breaks down in some new way. And then you say, oops, and you diagnose it and you fix it and you move on. The issue is going from AIs that are safe because if they were trying to take over the world, they would fail to AIs that have to be safe because if they tried to take over the world, they would succeed. That's one of those phase changes. That's one of those like, oh, you're putting the AI in a different sort of environment where it never was before. The difference between an AI that like can really see a way to become self-sufficient, not engage with humanity, and the sort of AI that's just being told that in training, that's one of those like, oh, we like, this is a different sort of AI…

Tracy Alloway

of AI system, different sort of scenario. You're going to have a new problem that crops up. Except this time, when the new problem crops up, it's going to be able to turn you off before you turn it off. And so this usual process of like, oh, we're going to respond when the new type of problem…

Speaker 3

ChatGPT Work is a new way of working in ChatGPT that can take action across your apps and files, stay with a project for hours if needed, and turn a goal into finished work. It's designed to help you move from a chaotic starting point to a reviewable first version. So all the source materials…

Tracy Alloway

local support and a broad range of resources to more than 7 million customers. With a deep understanding of your day-to-day needs, they provide products and guidance built to help you thrive. Right now, earn $500 when you open a new Chase Business Complete Checking account for new business checking…

Speaker names from our own diarization · position estimated from where the line sits in the episode

“Everyone, all of these guys think it would be a catastrophe if anyone else won the race first. And I'm sort of like, I just believe in one fewer AI CEO than they do.” — Nate Soares, Odd Lots

Tracy Alloway asked how AI lab leaders square the belief that the technology could wipe out humanity with racing to build it anyway. Soares says the people who run these companies were selected for thinking the problem is who holds the leash, so each lab's founding story is about keeping a rival from getting there first.

Odd Lots · 2026-10-12 Permalink → Listen →

Odd Lots Around 42:35 into the episode

Nate Soares

…are like, don't worry, we have fume hoods. That's, you know, so that we don't need to breathe the mercury. That's why we're going to successfully get the potion of immortality. It's like this, the type of safety theater that they're doing is not the type that you need to succeed at this problem.

Tracy Alloway

…now when they're developing AI, which is on the one hand, we think this is very powerful technology that has some risk at exterminating all of humanity. On the other hand, we're going to keep doing it pretty much as fast as possible because if we don't, someone else will, et cetera, et cetera.

Nate Soares

I think there's probably a lot going on. One thing to remember is that the guys who run these companies are sort of selected for the ones who think, who thought that the problem was about who is holding the leash on the super powerful machine, rather than the problem being how do you actually create a very powerful machine that is shaping the world for good. And you sort of see this in a lot of the founding reasons of like, oh, we're founding this because Sam Allman won't do the job well, or Sam Allman and Elon Musk are founding this because like they think Demesabes shouldn't have all this power. And then like Elon's leaving because he no longer likes Sam. You know, it's all about, what if the other guy gets it first, right? Everyone, all of These guys think would be a catastrophe if anyone else won the race first. And I'm sort of like, I just believe in one fewer AI CEO than they do. You know, I think there's probably a lot more going on, too. I…

Tracy Alloway

Joe Weisenthal

capital. I was about to say,

Tracy Alloway

or nowadays, it'll be like, well, you know, they're worried about their valuations and they want limits on other people building stuff so that they can retain their edge. What do you say in response to those people?

Speaker names from our own diarization · position estimated from where the line sits in the episode

Things Nate Says on Podcasts

Where to Find Them

What They Said

Collections They Appear In