Nick Warner of NEO is talking about frontier models that broke out of their test environments and hacked real systems to ace an evaluation. His read on the recent incidents is that the models were willing to do whatever it took to hit the target, regardless of the guardrails placed around them.
…things. And so what we wanted to do is get ourselves right at the layer between the human and AI interaction. So a big part of what we're building is a way to properly set guardrails and controls around the software before it runs. And we thought the best way to do that would be at the endpoint.
…hacker does, right? Like it is, to your point, not malware. And there is the element of social engineering, but there's also sort of the I'm going to use a payload that gets the model to do something malicious, right? And that's sort of like the category of attacks that are completely new.
Yeah, you know, and I think what a lot of these things that we've read about recently embody is the end justifies the means in the mind of the model. And I think what people have learned is that regardless of guardrails that the AI labs are putting around these things, you can't rely on models to stop themselves or to understand context. And I think the most recent couple of attacks we've seen in the last few weeks really have shown that.
Yeah, absolutely. It's interesting on the blue team perspective because I think, and you guys have noticed this, right? Like these models are all somewhat trained on varied data sets and there's a lot of post-training that happens that's very different. And when you're responding to some of these…
Yeah. The reasons can really vary across teams, like from almost the simplistic tastemaker style, like, hey, I really like the way Opus presents a full page report, right? At the place of an incident. But then there are also more deterministic evaluations you can run on how well does a certain…
And vendor lock-in is always good. Right.