A kill chain is the full sequence of steps in a successful intrusion; Mandia's company built a test from 20 that human operators had completed. He said the closed models were faster, but the open ones reached the same result when left to run longer. His conclusion was that the gap between open and closed models is smaller in cyber than in other fields.
…it's almost like a lunar landing race, right? The AI base. And you have R D people and they're doing the work to create models in a way where even those CEOs are like, we can't slow it. Let's get the government to help us slow it. That means you can't even control your own innovation. I have views
on that, but we need to get another time. And so
…time are made by the red teamers. You know what I mean? They're the ones that understand this stuff. And we created 20 full kill chains at Armadin that humans have done in the real world, period, at different victim sites and our experienced operators have done when testing networks. And we had no model go through the entire kill chains of more than eight. So that's where it was eight out of 20. And here's what's weird, by the way, we tested the open weight ones and the most advanced closed models. They all found eight. So if you're, yeah, so it was all about just speed and cost. And the closed models were faster to finding exploitable risk, but that's coming down. But we kind of let the open models run longer and they got to the same place. So in the cyber world, the differentiation between closed and open is not as great as in other domains, probably. And it's compressed. Yeah. Yeah. I would say that's…
consistent in terms of like the capability gap at least closing a little bit. But that's interesting that their performance is basically the same.
From my perspective, seeing the charts from the team, all the lines ended up in the same place. And when you're looking at, it was immediately time, cost, and then call it effectiveness or creativity. They all ended up at the end of all their operations where they hit diminishing returns, they…
not on cost. Yeah, that makes sense. Yeah, that makes sense. So it's a decent segue maybe to talk about what kind of models you guys are using. And then what role do you think the lab companies play in the future?