Sonwalkar was explaining what his company's new finance benchmark found about frontier models working on messy deal data rooms. Habits that make models strong at coding fail when the documents disagree with each other.
Right? And so one of the things we did with with this launch is we specifically did not measure the models on our own harness. We use the native CLI. So for open AI models, we use codex.
John Coogan
Sure.
Speaker 7
For Clot models, we use ClotCode. Yep. And just to be independent and really understand where the models are. Mhmm. And so that's the first step, is push with the curiosity and see if we can build the first ultra long horizon benchmark for finance workflows, and we did. And now what we're doing is understanding where do those different models stack up, what are the different idiosyncrasies in how these models pursue these tasks. Yeah. So one of the things we noticed is models that are really good at coding, you know, they tend to search a code base. Right? They didn't search a code base to understand it. That doesn't always work in finance. Yeah. If you just search for revenue in the documents Yep. And you see a number, you can't just assume that's the revenue. You have to see if it all adds up.
John Coogan
Yeah.
Speaker 7
And so...
John Coogan
Well, many many modern deal makers aren't even looking at the revenue. They just ask, like, who else is in over a text message, and then that's their criteria. So you you should fine tune a model on that. They just ask ask ask who's who's leading and then and that's the entire that's the entire…
Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
