“I think of this a little bit like if you have a bunch of alchemists trying to make a potion of immortality and some of them are just like breathing the mercury and other ones are like, don't worry, we have fume hoods. … It's like this, the type of safety theater that they're doing is not the type that you need to succeed at this problem.”
Odd Lots · Money & Markets · October 2026
Soares has just walked through incidents of AI models misbehaving outside their sandboxes, and the labs' responses to them. Anthropic, he says, has safety aspirations but nothing like the safety culture of aviation, nuclear power or NASA. The labs have a "veneer of safety" while racing as fast as they can.
Transcript
…of weak because we have actually seen cases with Claude in particular where we've seen Claude's chain of thought involve strained rationalizations. The sort of example paper here is there's a paper where you ask Claude, what's the total weight of drafts on the planet? And it's like 30,000 tons.
Drafts.
…But they admit that they have no idea what's going on inside their AI systems. They're racing straight for recursive self-improvement of like the AI is making smarter AIs. They are not treating this the way that NASA treats a rocket failure in terms of trying to diagnose exactly what went wrong and not have that happen again. And they're not sort of anywhere close to that failure level. Like I think of this a little bit like if you have a bunch of alchemists trying to make a potion of immortality and some of them are just like breathing the mercury and other ones are like, don't worry, we have fume hoods. That's, you know, so that we don't need to breathe the mercury. That's why we're going to successfully get the potion of immortality. It's like this, the type of safety theater that they're doing is not the type that you need to succeed at this problem.
I'm glad you brought up the alchemist analogy. What's your, I guess, psychological diagnosis for the tension that is seemingly going through every lab leader or company leader's head right now when they're developing AI, which is on the one hand, we think this is very powerful technology that has…
I think there's probably a lot going on. One thing to remember is that the guys who run these companies are sort of selected for the ones who think, who thought that the problem was about who is holding the leash on the super powerful machine, rather than the problem being how do you actually…
Let me ask a related question, which you hear a lot more about it now than I think you would have a year or certainly two years ago. But when people start talking about AI safety risks or the idea that it could kill us all, a lot of other people will say in response, this is just fear-mongering…
More from Odd Lots
