Connect your Claude, Codex or OpenClaw for full transcripts on millions of podcasts. Two week free trial, cancel anytime.

Start free trial →
SB Sahil Bhaiwala On Latent Space

“For this benchmark, for example, we interviewed hundreds of bankers. And one question we asked that I thought was really interesting is, would you rather start from this output or not? And the vast majority said that they would have rather started from scratch. And so this, I said, like, sit with that for a second.”

Latent Space · AI Builders · November 2026

“For this benchmark, for example, we interviewed hundreds of bankers. And one question we asked that I thought was really interesting is, would you rather start from this output or not? And the vast majority said that they would have rather started from scratch. And so this, I said, like, sit with that for a second.” — Sahil Bhaiwala, Latent Space

Context

The hosts argued that a mid-range benchmark score still carries useful signal for AI labs. Bhaiwala agrees from the lab side, but says that for real use in a bank, a partly correct AI-built model is worse than nothing.

Transcript

Latent Space Around 1:27:31 into the episode
Sahil Bhaiwala

think it depends on the perspective, right? I think

Speaker 3

it's totally correct.

Sahil Bhaiwala

But for this, for the for this benchmark, for example, we interviewed hundreds of bankers. And one question we asked that I thought was really interesting is, would you rather start from this output or not? And the vast majority said that they would have rather started from scratch. And so this, I said, like, sit with that for a second, right? Like the vast majority say they would have rather started from scratch. I think you're totally right from the lab perspective, right? Having the eval there to be like, hey, you know, we got to get better at surfacing these tax treatments or whatever, whatever ways those manifest. There's tons of those examples, right? It can certainly help you direct where do I need to improve steerability of the model? Where do I need to improve, you know, where might I need more training data? Where might I need, you know, you can figure out the solutions and solve for those. But if you're looking at it from the perspective of like, how do I get this used in a bank? You're talking a lot more about. Interpretability and like this human-agent interaction that starts to matter…

Speaker 2

of the normal checks that like generalize across all the cases. But this for this example task, this was the central emphasis anyway. So maybe just like the weighting of the numbers needs to shift over time. Should we broaden out or was there more in this? Was there less? I

Speaker 3

actually have a question to take this to make it more broad. You know, you've shown us the environment, the task, how complex it is. How do you come up with 93 checkpoints, 1400 points across these categories? Penalty traps, I think that's pretty intuitive. These are hard blockers, dependency gate…

Speaker 2

Yeah. So again, a big emphasis is making that as deterministic as possible, right? So our key kind of anchor is how do we how do we architect the task and the environment so that it is gradable on a rubric? It's a valid criteria. The sections are created by our experts with multiple layers of…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from Latent Space