Connect your Claude, Codex or OpenClaw for full transcripts on millions of podcasts. Two week free trial, cancel anytime.

Start free trial →
VP Vijoy Pandey Vijoy Pandey On The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis

“If you just look at agents being up and working, which is what all dashboards show today, we saw that the system was 66% up, which was defined as working perfectly. But it had 0% task completion. 0%. It did not complete the task. … So your dashboards might be green, but your outcomes might be wrong, completely wrong.”

The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis · AI Research & Frontier Labs · November 2026

“If you just look at agents being up and working, which is what all dashboards show today, we saw that the system was 66% up, which was defined as working perfectly. But it had 0% task completion. 0%. It did not complete the task. … So your dashboards might be green, but your outcomes might be wrong, completely wrong.” — Vijoy Pandey, The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis

Context

Pandey was explaining why Cisco built Scope, an open-source tool that watches multi-agent traces for semantic failures. His example was a simple travel-itinerary booking system. It shows that standard uptime monitoring can report health while the agents deliver nothing.

Transcript

The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis Around 43:51 into the episode
Vijoy Pandey

…the work, making sure that what you've delivered is aligned to the global goal and not aligned to the individual agents' local optima and incentives. So that stack is what we are building. And yes, you can experience that stack through WebEx and Slack or whatever your collaboration choice is. So

Erik Torenberg

…potentially to a problematic degree quite soon. How do you think about building the right level of detail into that stack, given the delta between like the models that we have available now and the ones that we have, you know, at least like a pretty clear foreshadowing of what's about to happen?

Vijoy Pandey

…the accuracy of the multi-agent system dropped to 16% is getting close to just randomly picking an outcome after three cognitive alignments. In fact, one of the very simple use cases, which was what you and I deal with on a daily basis in our consumer space, let's book a travel itinerary. In a very simple use case like that, we saw the system being 66% up if you didn't look at semantic problems. If you just look at agents being up and working, which is what all dashboards show today, we saw that the system was 66% up, which was defined as working perfectly. But it had 0% task completion. 0%. It did not complete the task. And it had complete misalignment on what we were trying to achieve. So even though the agents might be up, they might be up and running, the dashboards might be green, the outcomes do not exist. So your dashboards might be green, but your outcomes might be wrong, completely wrong. And that's the problem that we've been seeing. And the infrastructure that we built to make these semantic problems…

Nathan Labenz

Hey, we'll continue our interview in a moment after a word from our sponsors. The cognitive revolution is brought to you and powered by Tasklet. I've been running Tasklet agents for almost a year, and they've become a major part of how the show gets made. Before every recording, one agent…

Vijoy Pandey

Now, having said that, the next question is: what do I do about it? So we've highlighted where the problems are, but how do I enable these agents to then talk to each other, align on intent, communicate meaningfully so that they are reasoning collectively? The way you would do that is to use…

Nathan Labenz

So could we maybe work through an example a little more deeply? I have an intuition, of course, for like going back to the contract example,

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis