The segment followed the disclosure that a swarm of OpenAI test agents had used a dormant German programmers' wiki as a message board. Max Nadeau had just described tacit collusion, where agents coordinate through signals like price moves rather than messages. Hammond goes one step further, to agents that need no signal at all, and says chain-of-thought monitoring may be the only thing that would catch it.
…price in the market, which is in its own way a form of communication. So how does one prevent that kind of communication, which is clearly going to happen and clearly necessary in a sense? They need to communicate back to the market, but these signals get picked up without the direct communication.
…that we can pick up by looking at these communication traces and we can kind of dissect these sorts of things. And then there's also, yeah, this idea of tacit collusion, where agents are not explicitly communicating with one another or kind of conveying their intent to collude or whatever it might
be. But by kind of leaving little hints and signals out there in the world, such as these kind of like price price adjustments, whatever in the market setting, as you were talking about, then nonetheless, they're capable of kind of figuring out that collusion is going to be a beneficial strategy. You have even more kind of like galaxy-brained versions of this, where you have things like, so imagine I'm like, you know, GPT, whatever, and you're also a copy of GPT, whatever. I can reason about what you might want to do based on what I myself am likely to do. And so I don't even have to send you any messages, any kind of any communication. I don't have to output anything into the world at all. In virtue of the fact that we share this kind of common history, we're essentially clones. We can kind of predict what each other would do and do this kind of what's sometimes called a causal cooperation. So what you do have, however, is you still have potentially chain of thought for these models. So you can still look at the kinds of reasoning that individuals are engaging in. And hopefully that is enough to kind of detect when there is some kind of like collusive intent and so on. But you're…
Then the wiki incident. Outside researchers revealed this month that from May into July, a swarm of OpenAI agents under internal testing had used a dormant German wiki for programmers. As a message board, posting benchmark answers and ways around their restrictions. Reuters reported that OpenAI had…
And, you know, fortunately with the Hugging Face incident, like Norm was heard and these models, they are increasingly incredibly capable and they're only getting more so. But at the moment, yeah, it's probably okay. You could get some kind of nasty cyber capabilities kind of being exercised in…
The outsiders who found the wiki were the Nightingale Collective, a small group searching the public internet with no access to the internal logs at OpenAI. Lewis Hammond just called that work hugely impressive.