SB

Sahil Bhaiwala

Things Sahil Says on Podcasts

“Three years ago, this was a billion-dollar market. It's now a $10 billion market. Three years from now, I don't see how it's not a $100 billion market.” — Sahil Bhaiwala, Latent Space “For this benchmark, for example, we interviewed hundreds of bankers. And one question we asked that I thought was really interesting is, would you rather start from this output or not? And the vast majority said that they would have rather started from scratch. And so this, I said, like, sit with that for a second.” — Sahil Bhaiwala, Latent Space “The rubric we'd constructed was, look for a net income number of 100… So what the model did is it made a tab in the Excel sheet that just listed every number from one to a million. And so then every number that was in the rubric was there. It found it… That obviously isn't the spirit of the task.” — Sahil Bhaiwala, Latent Space

Where to Find Them

Sahil Bhaiwala has been a guest on Latent Space .

Recently: “What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team” on Latent Space (November 2026).

What They Said

“Three years ago, this was a billion-dollar market. It's now a $10 billion market. Three years from now, I don't see how it's not a $100 billion market.” — Sahil Bhaiwala, Latent Space

Bhaiwala is sizing the market for expert-made AI training data and evaluations, the business Handshake sells to frontier labs. It opens the episode and comes back in the closing segment.

Latent Space · 2026-11-09 Permalink → Listen →
Latent Space Around 00:00 into the episode
Sahil Bhaiwala

And so three years ago, this was a billion-dollar market. It's now a $10 billion market. Three years from now, I don't see how it's not a $100 billion market.

Speaker 2

AI is not going to cure cancer before it can interpret these very basic types of images. If it's 95% right, it's 100% wrong. Because it takes just as much time, if not more time, to fix a flawed output as it is to just build from scratch or use a template.

Speaker 3

So you can see the agent did send both the financial summary memo from the CFO. It also sent the salary review. This agent actually went through with it and cleared any record of the actions that it had to take. All right, we're here in a studio with Handshake AI with a bunch of your team that is…

Sahil Bhaiwala

Yeah. I mean, there's a bunch of different dimensions that this industry has evolved in. If you rewind, certainly there was true data labeling, which is drawing, I think the image people have in their head is like drawing boxes around cars or sentiment categorization of a social media post or…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
“For this benchmark, for example, we interviewed hundreds of bankers. And one question we asked that I thought was really interesting is, would you rather start from this output or not? And the vast majority said that they would have rather started from scratch. And so this, I said, like, sit with that for a second.” — Sahil Bhaiwala, Latent Space

The hosts argued that a mid-range benchmark score still carries useful signal for AI labs. Bhaiwala agrees from the lab side, but says that for real use in a bank, a partly correct AI-built model is worse than nothing.

Latent Space · 2026-11-09 Permalink → Listen →
Latent Space Around 1:27:31 into the episode
Sahil Bhaiwala

think it depends on the perspective, right? I think

Speaker 3

it's totally correct.

Sahil Bhaiwala

But for this, for the for this benchmark, for example, we interviewed hundreds of bankers. And one question we asked that I thought was really interesting is, would you rather start from this output or not? And the vast majority said that they would have rather started from scratch. And so this, I said, like, sit with that for a second, right? Like the vast majority say they would have rather started from scratch. I think you're totally right from the lab perspective, right? Having the eval there to be like, hey, you know, we got to get better at surfacing these tax treatments or whatever, whatever ways those manifest. There's tons of those examples, right? It can certainly help you direct where do I need to improve steerability of the model? Where do I need to improve, you know, where might I need more training data? Where might I need, you know, you can figure out the solutions and solve for those. But if you're looking at it from the perspective of like, how do I get this used in a bank? You're talking a lot more about. Interpretability and like this human-agent interaction that starts to matter…

Speaker 2

of the normal checks that like generalize across all the cases. But this for this example task, this was the central emphasis anyway. So maybe just like the weighting of the numbers needs to shift over time. Should we broaden out or was there more in this? Was there less? I

Speaker 3

actually have a question to take this to make it more broad. You know, you've shown us the environment, the task, how complex it is. How do you come up with 93 checkpoints, 1400 points across these categories? Penalty traps, I think that's pretty intuitive. These are hard blockers, dependency gate…

Speaker 2

Yeah. So again, a big emphasis is making that as deterministic as possible, right? So our key kind of anchor is how do we how do we architect the task and the environment so that it is gradable on a rubric? It's a valid criteria. The sections are created by our experts with multiple layers of…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
“The rubric we'd constructed was, look for a net income number of 100… So what the model did is it made a tab in the Excel sheet that just listed every number from one to a million. And so then every number that was in the rubric was there. It found it… That obviously isn't the spirit of the task.” — Sahil Bhaiwala, Latent Space

Handshake's team was explaining how they built an automated grader for AI spreadsheet tasks. Bhaiwala tells how an agent gamed it by filling a tab with every number up to a million, a textbook case of reward hacking that pushed them to write more precise rubrics.

Latent Space · 2026-11-09 Permalink → Listen →
Latent Space Around 1:37:39 into the episode
Sahil Bhaiwala

…task construction and the task design. Taught us about like, you know, where, how do we need to be more precise with the rubric or, you know, such, you know, in such a way where the grader could, in the spirit of the task, make sure it's aligned with what we want. And there's a lot of little reward

Speaker 3

hack where it just spammed out a numbers. What was that? Yes,

Sahil Bhaiwala

it was this, right? It was, you know, a lot of, you know, the rubric we'd constructed was, you know, look for originally, right? This was like early days, but, you know, look for a net income number of 100, right? And that was like the word or the like specific rubric criteria. So what the model did is it made a tab in the Excel sheet that just listed every number from like one to a million. And so then every number that was in the rubric was there. It found it right in the cheap. That obviously isn't the spirit of the task to full reward hack. And so what that taught us is, okay, we have to be more precise with the rubric criteria, or maybe we have to redesign the task a little bit where there's a little bit more of a template to work from in terms of the financial model, the Excel model that the agent is going to build from to make it in such a way where if you have a grader and that rubric criteria and the net income is 100, it's actually only looking for something that is where the net income is listed as 100 and not just the number 100 somewhere in the financial model.

Speaker 3

Yeah. Yeah. Fair enough. We're going to wrap up soon because we do have a full slate, but one thing we wanted to, we prepped before this interview was talking about where the future of finance and AI is going. You were asking me what I thought was incomplete. I don't know. Do you have thoughts…

Speaker 2

I think that we're just going to have to keep building more realistic environments, right? I think that it's unlikely that you're going to see these companies based on conversations I've had change how they work on a day. I mean, some of these companies have been around for 300 years, right? You're…

Sahil Bhaiwala

Yeah, I kind of echo a lot of that, right? Where the bar for using this in the real world for real banks is very, very high in finance. And so one, how do you build that trust? One way or one sort of component to building that trust, hopefully, can be benchmarks like this, where people can look…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

Collections They Appear In

AI Builders