From the cold open of a highlights episode. Arena's CEO describes what an audit of real production conversations found: misaligned behavior grows sharply with conversation length.
This week on AI in the AM, how much of what we think we know about AI survives contact with the data? Anastasios Angelopoulos, co-founder and CEO of Arena, the leaderboard formerly known as LM Arena, which now audits real conversations for misaligned behavior. Here's what its production traces show.
If you look at the first message in the conversation, the misalignment rate is relatively low. It's in the single-digit percentage, which is, you know, to be fair, that's actually not nothing, but it's relatively low. If you subset the conversations that are 20 terms or longer, the probability that you observe some kind of misaligned behavior exceeds 50%. It's actually insane. What that means is that every single one of us, if you're a heavy user of AI at least, has had at least some conversations where the AI has been lying to us, misleading us, or touching, accessing data on files that it shouldn't have. And, you know, we might not even know it.
Gemini 4 Carbon, Google's new model, landed on Arena's leaderboard while we were live on a stages on first look.
It's so good. Google is back. Google is back in a big way. You know, Carbon is, Carbon puts Google the number two lab in the world on our agent arena. Huge result. The agent arena is a very challenging benchmark, as they do with all of these like open world agentic capabilities. And it is number…
Debo Olaosebikan Oshevikin, co-founder and CEO of Kepler Computing, which is building a new kind of memory for AI chips in older fabs. I asked how the compute forecasts should change if memory moves off the scarcest tools.