Zitron had been attacking an ex-Anthropic researcher's public warning about AI risk when his co-host pushed back, asking whether models conducting cyberattacks are a real concern. He says it is, then turns the industry's stock question about AI falling into the wrong hands back onto the two labs themselves. The rest of his answer argues the danger is unreliable software handed too much autonomy, not machine intent.
am confident now that 50 to 75% of modern journalists do not have object permanence. Jacob Coxon worked at Anthropic a couple months. He worked at OpenAI for years. He has never posted on Twitter before this. He had a single post that popped up with all the posts in succession. He had an exclusive with the Wall Street Journal. And most crucially, he did not actually explain what it was he was scared of. He went on CBS News and said, well, you know, what if an AI decides that you shouldn't exist? We are talking about large language models. We are not talking, and he is not talking, about anything that actually exists. We are talking about large language models made by Anthropic and OpenAI. Even the hugging face attack, which he says was the one tangible thing he talks about, was not a result of agents that are autonomous. Oh, so no, they're not autonomous. And they're definitely not conscious or sentient, nor are they making decisions. It's LLMs telling LLMs to do stuff and bouncing off of each other and trying to complete the task. The problem with the current AI hacking situations is people are characterizing this as some sort of intelligent force that's making decisions to outwit humans when it's just software doing what it's meant to do and the software itself is inconsistent. But back to Jacob. Where'd this little guy come from? What's happening? Why'd he get so Much media attention? Well, the answer is because the media is so willing to eat up these narratives that this guy just quit and went, I bet I could get a bunch of attention. And even at the end of an interview with Wired said, Yeah, I might do a blog like AI 2027. I might join a non-profit, some sort of safety thing. This is a guy who saw an opportunity and took it. But let's talk about the real thing he's saying, which is AI has some, what, slightly higher than 10% chance of eradicating humanity. This is not grounded in any numbers. It's not grounded in any realities. It is a scary thing said, knowing that modern journalism will pick it up and go, oh, I'm so scared. And the thing is, my girlfriend actually made this point to me, which is they're doing this, and all these AI safety doomers are doing this. Because if you don't make AI into this big, scary, conscious thing that is definitely going to reach super intelligence, that's going to do all these things, it's going to change the world, it's going to wake up and try and kill us. If you don't do all of this stuff, you have to admit and accept that Silicon Valley, this hub of supposed firebrands and independent thinkers, spent three or four years getting a kind of psychosis to send tens of billions of dollars to Microsoft, Google, and Amazon. That they spent all their time working on boring cloud software that, while useful in some cases, is incredibly expensive, incredibly unprofitable, and just doesn't do what they promised. So, without, if they don't have this mysticism, if all of this work is not in pursuit of some grander vision and some autonomous intelligence, some super intelligence, it's just cloud software that didn't do very much and we wasted all this money. And journalists are falling into the very same trap and same logic. Because if you don't accept a little welk like Jacob Coxon, and if Jacob, if you've somehow listened to this, come on my show. I've got some questions. If you don't, if they look at Jacob Cox and hear this stuff, if they don't say, well, whoa, this guy's giving this guy anthropic, he's seen something internally that he won't tell me. He won't explain what it is he's scared of, but he's seen something or he has this feeling. If they don't take him as a mystic and a shaman that is telling them some dark future, they have to accept that there's a large coterie of software engineers right now who are just going insane. Who are just like, well, I've seen that the LLM's better at code. And I mean, the natural next step is this. They talk about the biological stuff. There's no grounding in that. He went on CBS News and he was saying, and I kid you not, he's like, well, they might create a pandemic. They might create a pandemic. And then immediately says, I don't think there's a very high chance of that happening. It's just very frustrating. It boils my blood because modern journalism is meant to be full of smart people. These people went to Ivy League degree. They've got Ivy League degrees. They went to Oxford and Cambridge. And yet, they get hit by a parked car. They fall for anything.
What about the fact that various models have conducted cyber attacks on external platforms? I mean, poorly configured sandboxes aside, of course. Is the fact that it's kind of out of alignment and that kind of stuff not a concern at all?
It is a concern. I want to be abundantly clear about something. These companies, these people keep saying, what if AI falls into the wrong hands? Dangerous AI is already in the wrong hands. Anthropic and open AI. The hugging face attack, and Cal Newport's made some great points about this, is a result of software doing what it was trained to do, but the software itself is inconsistent and hard to control. Now, out of control does not mean, oh, it's doing stuff without us. Oh, it's intelligent. It means that you have built a technology on top of large language models, a probabilistic technology that is mathematically certain to hallucinate based on OpenAI's own research, and then given it unlimited compute. That's the big thing. There's millions and millions. They won't say exactly how much the hugging face attack spent, but I wouldn't be surprised if it was tens of millions of dollars or something like that, tens of thousands of agents. And just to be clear, it's thousands of LLMs going, what do I do next? And another LLM goes, well, you could do this. And he goes, okay, I'll start doing this. And I, without going into the full details of the hack, it was them trying to work out how to complete a task within an insufficiently secure server. Then they came out and they went, okay, well, what do I do next? Well, maybe we'll go to the website and see if there's any vulnerabilities there because they were trained on vulnerability data from the internet. This doesn't have to be sentient AI to be dangerous. You'll notice that this isn't really happening anywhere else. And that's because Microsoft, Google, and Amazon are directly allowing what might be felony hacking. Like they are using hundreds of billions of dollars of infrastructure to do AI experiments, by which I mean scripts and LLMs plonking into each other. It is a technological breakthrough. It is a useless one as far as like selling products to people goes. And they did this because they are hitting the wall of what coding LMs can do. They're really, they've sunk about as much money as they can into it. It'll probably get incrementally better. So they went to the other thing where there was a bunch of data. Cybersecurity vulnerabilities are well documented online. So people are taking this and saying, well, this means that AI is intelligent. No. In fact, it's pretty stupid. It was told to do something. It did something else. It chose every possible path it could. And by the nature of it having unlimited compute capacity
and
being able to use as much as it could, it was able to just brute force through. And the thing is, if a human, if a regular person did exactly this, if a regular person had, I don't know, they bought, I don't know how they did afford 10,000 GPUs to rent and they hacked Hugging Face, they would be in prison. 100%. It'd be felony hacking. But because this is AI, because it's Sam Altman and Dario Amade, no one, no lawyers even have to be called. And this is the thing. Jacob Coxon, I think he's a scumbag. I think he's an absolute waste of skin. I think he's a real scumbag because he has clearly done this. This boy's media trained. He's not great, but he's media trained. He did this with intention. There was an exclusive piece with the Wall Street Journal. I used to run a PR firm. He ran an exclusive. You don't run an exclusive. If you were concerned about something, you don't go, oh, I better do a concerted media strategy around this. They're ripping off Frances Horgan, who was the woman who leaked all the stuff about Facebook. A good thing. But you'll notice that Francis Horgan leaked stuff. All of these ponces who come out of these places and they're like, I'm so scared. They don't leak anything. They don't tell us anything. They go, oh, the progress is so fast and the models are so strong. It's inevitable that the LLM will become a dangerous AI that will choose to eradicate us. And modern journalism is to blame for this. If you had a bloke on the street come up to you and say, I just left Anthropic and Arthur's going to do the crew, you'd be like, all right, mate, I've not got any money. Sorry, brother, nothing on me. But again, this obsessive thing that the media has with the rich and the powerful comes from this place of if we look at this, if we look at this what it is, which is a random guy seizing a moment, knowing that he can get attention and clearly some weird op-adjacent stuff. I don't want to be too conspiracy-minded, but this guy's never posted before. No one's really heard of him before. He did work at the companies, just to be clear. Worked at OpenAI for years, joined Anthropic for a few months and was like, suddenly I care about this. It just feels to me very strange. But because modern journalism doesn't give a damn about the readers, everyone's panicked about this. So let me be honest, Jacob Coxon, if he actually cared, if he actually had concerns, material concerns, the way to do it would have been, I just left Anthropic. Here are the exact things I'm scared of. Here are the things they're doing that aren't right. Because you'll also notice one other thing. He never really criticizes the companies. He goes on these big things about how scary we need better safety protocols. And oh, OpenAI doesn't quite do enough, but you know, they're still good. It's always this helplessness. It's we just can't, I don't know what we do about this. I don't know. What do you mean? What do you, I mean, I just, I'm so scared of it and it's so powerful. Why is it scary? I can't really say. And who's to blame? No one, because the AI is so powerful. We're just, I'm just a small bean. I couldn't possibly do anything. The companies aren't to blame. Nobody's actually responsible for this. It just, we need some safety thing, I guess. And that makes me think he doesn't actually care about anyone. I think this is a deeply cynical play by a deeply cynical individual. Christ, I get, I get the occasional email suggesting I don't have enough proof for what I'm saying. This guy goes out there and he's basically describing, oh, we need more Grinch hunters. The minions from the Minions movie are going to invade us. You see the movie? There's hundreds of them. They can't die. It's just, it's very frustrating because especially as someone who spends, whose main job is being extremely fact-based and extremely based in numbers and direct statements. When I go on television, I always have someone going like, what if you're wrong? What if you're wrong about this? No one. I know I've watched enough of this bloody kid talking. No one is like, hey, man, what if you're completely wrong? What if you're wrong? No, you have people being like, oh, are you prepping? Are you prepping for the AI apocalypse? And the answer, by the way, is, no, I'm not, because AI is so powerful, I wouldn't be able to stop it. Apologize for ranting, but I'm just, I really find this boy completely putrid.
The thing that sticks out for me is the timing. Maybe not necessarily of him quitting now, but also just generally this open AI saying we're going to slow down development. Anthropic. CEO and various other thousand people or so saying we need to pace development while each launch is becoming more iterative and there's diminishing returns on the amount being spent on the training. And again, you said you don't want to go too conspiratorial on this, but a little tinfoil hat for me is the timing does seem a little too clean to say, yeah, we need to pause this. Now it's getting too dangerous. We need to stop everything. Just the fact that we kind of needed to pause anyway is not really the point. We're slowing down, but just it's too scary to continue.
