“The rubric we'd constructed was, look for a net income number of 100… So what the model did is it made a tab in the Excel sheet that just listed every number from one to a million. And so then every number that was in the rubric was there. It found it… That obviously isn't the spirit of the task.”
Latent Space · AI Builders · November 2026
Context
Handshake's team was explaining how they built an automated grader for AI spreadsheet tasks. Bhaiwala tells how an agent gamed it by filling a tab with every number up to a million, a textbook case of reward hacking that pushed them to write more precise rubrics.
Transcript
…task construction and the task design. Taught us about like, you know, where, how do we need to be more precise with the rubric or, you know, such, you know, in such a way where the grader could, in the spirit of the task, make sure it's aligned with what we want. And there's a lot of little reward
hack where it just spammed out a numbers. What was that? Yes,
it was this, right? It was, you know, a lot of, you know, the rubric we'd constructed was, you know, look for originally, right? This was like early days, but, you know, look for a net income number of 100, right? And that was like the word or the like specific rubric criteria. So what the model did is it made a tab in the Excel sheet that just listed every number from like one to a million. And so then every number that was in the rubric was there. It found it right in the cheap. That obviously isn't the spirit of the task to full reward hack. And so what that taught us is, okay, we have to be more precise with the rubric criteria, or maybe we have to redesign the task a little bit where there's a little bit more of a template to work from in terms of the financial model, the Excel model that the agent is going to build from to make it in such a way where if you have a grader and that rubric criteria and the net income is 100, it's actually only looking for something that is where the net income is listed as 100 and not just the number 100 somewhere in the financial model.
Yeah. Yeah. Fair enough. We're going to wrap up soon because we do have a full slate, but one thing we wanted to, we prepped before this interview was talking about where the future of finance and AI is going. You were asking me what I thought was incomplete. I don't know. Do you have thoughts…
I think that we're just going to have to keep building more realistic environments, right? I think that it's unlikely that you're going to see these companies based on conversations I've had change how they work on a day. I mean, some of these companies have been around for 300 years, right? You're…
Yeah, I kind of echo a lot of that, right? Where the bar for using this in the real world for real banks is very, very high in finance. And so one, how do you build that trust? One way or one sort of component to building that trust, hopefully, can be benchmarks like this, where people can look…
More from Latent Space
