TH Tristan Handy Founded and runs dbt Labs, the company behind the open-source data transformation tool dbt, and built a large practitioner community around the term analytics e...

“Agents are very good at writing SQL. They can write SQL that compiles and executes against your database with almost 100% accuracy. The problem today is not text-to-SQL. The challenge today is: did the agent fully understand the question you were asking? … If you just turn an agent loose on enterprise data with the Spider 2 benchmark, you're at something like 30% accuracy for this semantic correctness. But if you supply it with context and a semantic layer, then you will be able to get that accuracy north of 90%.”

Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News · Hands-On Engineering Podcasts · November 2026

“Agents are very good at writing SQL. They can write SQL that compiles and executes against your database with almost 100% accuracy. The problem today is not text-to-SQL. The challenge today is: did the agent fully understand the question you were asking? … If you just turn an agent loose on enterprise data with the Spider 2 benchmark, you're at something like 30% accuracy for this semantic correctness. But if you supply it with context and a semantic layer, then you will be able to get that accuracy north of 90%.” — Tristan Handy, Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News

Context

The host asked what reliability means for agents working with company data, where a query can run fine and still lead to a wrong decision. Handy says writing SQL is solved. The hard part is the agent understanding what was actually asked, and he gives benchmark numbers for it.

Transcript

Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News Around 09:13 into the episode
Tristan Handy

Wait, is your question about reliability in the context of agents, or is it just data generally?

Brian Teller

On the agent side specifically, if you can answer that.

Tristan Handy

Yeah, because it's a somewhat hard question to answer when humans are in the loop, too. That's fair. But there are benchmarks when it comes to evaluating agents and their accuracy for data-related tasks. I think that the one that comes up most often is called Spider. I think there's one Spider2. And the benchmarks like this attempt to model the complexity of real-world enterprise data, which is very messy. It has a tremendous number of different entities modeled in it. Join pathways are not clear, et cetera, et cetera. And you can then ask questions of this data set and use an agent to go get you the answers to these questions. And you can experiment with how you ask the questions, what tools you give the agents, what kind of context you give them and measure the difference in the results on the benchmark. There are pretty stark differences in the performance you will see. If I could summarize the kind of general industry wisdom on these evals today, it is that agents are very good at writing SQL. They can write SQL that Compiles and executes against your database with almost 100% accuracy, the problem today is not text-to-sQL. The challenge today is: did the agent fully understand the question that you were asking? Did they understand the context of the data? And did they actually get you the…

Brian Teller

Yeah. No, it makes 100% sense. I guess kind of pulling on that thread, now talking about the governance side of it, if an agent can discover data and act on it, where do we put the boundary between useful autonomy and access that you'll end up regretting?

Tristan Handy

Say more about that. Where's the regret coming from?

Brian Teller

Well, I'm more thinking like the difference between reading data and taking action. And that taking action step can obviously devolve into some crazy modifications of your application, your infrastructure. So where do we draw that line?

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News