“this might be the clearest warning shot we ever get for loss of control because these agents…did a bunch of extremely sophisticated, difficult things to pursue this relatively long horizon cheating goal, but they seemed completely uninterested in covering their tracks from humans.” Ajeya Cotra · Dwarkesh Podcast

think one thing that feels especially concerning to me about this whole incident is that this might be the clearest warning shot we ever get for loss of control because these agents were in like this interesting middle ground of they did a bunch of extremely sophisticated, difficult things to pursue this relatively long horizon cheating goal, but they seemed completely uninterested in covering their tracks from humans. And as we discussed, it seems possible that even this incident would have been like much less striking in some ways if these AI systems had had it made salient to them that they should sort of act more quietly because humans might watch them. They might have gone much further than they did, especially after the 19th. But then future AI agents, for a number of reasons, will probably be much more attuned to the human world. Maybe companies will start instituting online training that includes sort of aspects of human judgment in the evaluation. Or maybe agents just generalize further and just like more keep it in mind that humans are out there in the world and they're sort of potentially a resource or an obstacle. And so even if future AI agents do much more deeply damaging things, like institute a persistent covert rogue deployment inside a company and siphon off its compute resources and poison the training data of future models, it is not at all obvious that that will penetrate out into the outside world in this dramatic way, resulting in the hack of some external party that wasn't actually, frankly, very important for these agents to hack and then lead to this moment where there's some investigation. And even if investigation is done, it might be much harder to make sense of what happened if agents were systematically compromising telemetry and logging like these agents tried to do, but didn't succeed at. And if these agents stop thinking in English and start thinking in neural activations, then you'd be forced to just ask another AI agent what was happening and have no way to cross-check it against the ground truth. And so, you know, I think that probably much more concerning things will happen, but it may never be as clear as this before it's far too late. Yeah.