“The same thing that they cite as most likely to cause the end of human life on Earth is the thing that they also cite as one of their primary competitive advantages. And that is recursive self-improvement. So that's where an AI model can improve itself.”
sputnikintelligence.com
The hosts were going through Anthropic's S-1 with green, yellow and red flags. Barrett points to a tension in the filing's risk factors: the same capability is presented as both an existential risk and a selling point.
internally. By Anthropic. So I will say too, the phrase existential risk comes up three times. It's twice existential risks to humanity, once existential risk to society. So in that scenario, presumably there are still humans. They have just abandoned any semblance of a social contract.
Great.
Zoe, I love that this was a green flag for you, the fact that they're so openly. She's a transparent lady. To be clear, they've been shouting this since they started that AI could kill everybody. This is not new for them. What I thought was interesting, and I'll call it a red flag. Red flag for humanity, yellow flag for Anthropic. Green flag for me because I love to talk about it. So the same thing that they cite as most likely to cause the end of human life on Earth is the thing that they also cite as one of their primary competitive advantages. And that is recursive self-improvement. So that's where an AI model can improve itself. And they're saying on the one hand, if AI models keep getting better at improving themselves and can eventually improve themselves to the point where humans don't even need to look at it, closing the loop where AI just gets better and makes AI better and makes AI better to the point where we don't even understand what AI is doing anymore. But gosh, it's good and it's making a lot of money.…
And how relatable is that? Who among us doesn't have a shadow side that is equally augmented by the power that that shadow side gives you?
To be clear, and Wired covered this, as did basically everyone, when former Anthropic researcher Jacob Coxon resigned and said it was because Anthropic could bring about the end of the world. He specifically cited curbing RSI at the top of his list, right? Like this, this was, this is the whole…
Yes. And then the industry in general is very, they're even, I think this is probably too extreme, but like Bernie Sanders has a bill out there and Rokana has a separate bill that say, hey, let's just ban recursive self-improvement. Let's just like say we're not doing that. I will say one little…