Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News · Hands-On Engineering Podcasts · November 2026
The host asked what reliability means for agents working with company data, where a query can run fine and still lead to a wrong decision. Handy says writing SQL is solved. The hard part is the agent understanding what was actually asked, and he gives benchmark numbers for it.
Wait, is your question about reliability in the context of agents, or is it just data generally?
On the agent side specifically, if you can answer that.
Yeah, because it's a somewhat hard question to answer when humans are in the loop, too. That's fair. But there are benchmarks when it comes to evaluating agents and their accuracy for data-related tasks. I think that the one that comes up most often is called Spider. I think there's one Spider2. And the benchmarks like this attempt to model the complexity of real-world enterprise data, which is very messy. It has a tremendous number of different entities modeled in it. Join pathways are not clear, et cetera, et cetera. And you can then ask questions of this data set and use an agent to go get you the answers to these questions. And you can experiment with how you ask the questions, what tools you give the agents, what kind of context you give them and measure the difference in the results on the benchmark. There are pretty stark differences in the performance you will see. If I could summarize the kind of general industry wisdom on these evals today, it is that agents are very good at writing SQL. They can write SQL that Compiles and executes against your database with almost 100% accuracy, the problem today is not text-to-sQL. The challenge today is: did the agent fully understand the question that you were asking? Did they understand the context of the data? And did they actually get you the…
Yeah. No, it makes 100% sense. I guess kind of pulling on that thread, now talking about the governance side of it, if an agent can discover data and act on it, where do we put the boundary between useful autonomy and access that you'll end up regretting?
Say more about that. Where's the regret coming from?
Well, I'm more thinking like the difference between reading data and taking action. And that taking action step can obviously devolve into some crazy modifications of your application, your infrastructure. So where do we draw that line?