Pandey's team analyzed multi-agent sessions from its own deployments and from public traces, and found at least one cognitive failure in every session. Here he gives the numbers on how quickly accuracy collapses as failures stack up; he adds that 16% is as good as picking an outcome at random.
…in all of these vertical spaces that we've deployed them in. But we also took public traces from chat dev sessions, from MMLU probes that exist out there, and we analyze them. And what we found was pretty surprising. We found that 100% of all multi-agent sessions had at least one cognitive failure.
Oh, wow.
We found that these failures compounded. So if there was one failure, there was still 98% accuracy in the multi-agent system output. You go to two failures and it drops to 83%. When you go to three failures, it drops to 16% accuracy.
16, 1, 6.
1, 6. That's as good as randomly picking an outcome. And so these failures are, again, you don't see them because your observability dashboards are green. But these are silent failures that are going to impact the outcome that you're looking for from a multi-agent system.
I want to hear more about these tests and how you evaluated them in a second. I just, I want to clarify quickly for our listeners what SWE and SRI is. Probably most listeners are aware, but so SWI is software engineer. It's a TLA, a three-letter acronym for software engineer. And then SRI is SRE…