Zhou describes a benchmark in which frontier AI agents each had a week and a data budget to forecast the 2026 midterms. He singles out Meta's Muse models for phoning it in, and later suggests the harness design ends their runs early.
…And yet, there's just so much momentum, and it seems like they're very determined to do a version of it, at least, that still seems to me to be like pretty risky or certainly a lot riskier than I would personally endorse. Hey, we'll continue our interview in a moment after a word from our sponsors.
…that actually. Scale. Stop chasing the hype and start owning your agentic future. You can see how it works and learn more at outsystems.com/slash TCR. That's outsystems.com/slash TCR. Part two, Agents Forecasting, Agents Training. Our first Friday guest, Brian Joe, a researcher at Thoughtful.
…they think the margin of victory will be. For example, if the Democratic candidate will win by three points versus seven points, et cetera. So agents are tasked with giving incredibly granular and detailed forecasts. And there's a lot of really interesting things that we can and have observed from this. It's a really interesting test of long-running agentic behavior. In particular, for example, what we see with some of the Muse Wonder series models is that they're often incredibly bad at doing long horizon work. When given a task in the Muse code harness, when they have a list of, say, 800 or so elections that they need to predict, they'll create a random forecast, which is just a random skew off of the 2024 election results, and submit that within, for example, the first two, three minutes of working. I'm already intrigued by your description of the Muse agent, and it's seemingly phoning it in. So I guess one initial question just on that point is like, how do you reconcile what you saw with what people seem to be reporting online, which is like overall pretty good experiences? No,
for sure. I mean, our initial theory on this, right, is that this is mostly due to a part of the harness design. So we're not talking about Meta's sort of like personal agent. We're talking about their coding agent. And one of the things that we kind of noticed is the design of the Muse Wonder…
when paired with Muse code, which is the native harness that Meta provides, has essentially what is a task list. So even though in the prompt, for example, they're instructed, hey, you have seven days of time, you have unlimited API usage, and you have full permission to like fan out sub-agents, do…
So tell me about the results. How did the models compare against each other? How did they compare against top humans? Do we now have superhuman forecasters?