The host asked what a data team should do first when asked to connect agents to company data. Handy says the old best practices still matter. The one genuinely new piece is a context layer.
Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News · 2026-11-16
Permalink →
Listen →
Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News
Around 27:40 into the episode
Tristan Handy
…They don't ask you to move into a different version of the software. They plug into what you're already using. And rather than adopting some kind of enterprise version, you say, I want DBT state or I want late compute or whatever. And you can, on a consumption basis, pay for that specific thing.
Brian Teller
Okay. Yeah, that makes sense. I was, in my mind, I'm thinking about it going back to that Terraform example, HashiCorp IBM, you know, acquisition and how that kind of changed licensing. And yeah. So, yeah, yeah.
Tristan Handy
Pretty different. Yeah. No, no, no. We've only become more permissive with licenses since the merger. And we and Fytran are super different than HashiCorp and IBM in that both companies kind of grew up together and are directly complementary pieces of technology. And so they together form a very nicely packaged platform offering. Like
Brian Teller
symbiotic relationship more than, yeah, gotcha. Okay. So if I run a platform or data team and people are asking me to connect agents to our enterprise data, what is the first thing you'd have me do?
Tristan Handy
Oh my goodness. Well, the first thing is actually eat your spinach. All of the things that were good ideas in the past are still good ideas. If you have a well-governed data lake and the data in it is well modeled and you have a semantic layer, then and these are all things that like have been…
Brian Teller
Yeah, now that makes sense. Okay, so wrapping up, we were talking before we started recording. I'm just curious, what have you been building lately with AI tools?
Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
Asked how the analytics engineer's job changes when agents write most of the SQL, Handy says the same shift is coming for SREs and software engineers. The valuable work moves to designing the loops and evals the agents run in.
Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News · 2026-11-16
Permalink →
Listen →
Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News
Around 15:56 into the episode
Tristan Handy
…typically you move more slowly. And that final check is done in the same way that when you build a software factory today, you have this decision to make who merges the pull requests. And some people say the human does, and some people say that the agent does. And so you get to make that decision.
Brian Teller
Yeah. Okay, so let's move on to like analytics engineering. So DBT helped make data transformation look more like software engineering. AI Agents write more SQL and consume more data, how does the analytics engineer's job change?
Tristan Handy
My general guidance here is really not about analytics engineers or SREs or software engineers. It is all of these very technology knowledge work fields are going to be systematized in a way that they haven't been before. And you as a professional will either be operating inside the system or you'll be operating on the system. And from my perspective, the more interesting work to do is to be operating on the system, to be designing the loops, the evals, et cetera, as opposed to having tickets assigned to you. I think that this is true of all of the different types of jobs that I just mentioned and many more. And so for analytics engineers, that really looks like not how do I really cleverly model this data to solve this kind of thorny quantitative problem. It turns out that Astra 6 is going to be able to solve that problem as well as you can, if not better. And instead, you should be thinking about the system holistically and saying, what evals do I have? What evals do I need? How do I see the things that are unseen, like the latent business needs in the…
Brian Teller
Yeah, so it's moving up a layer, essentially. 100%.
Tristan Handy
One of our values that I now authored 10 years ago is we believe in moving up the stack. And I think this is a big moving up the stack moment. We've all been very identified with our own set of technical skills because we've spent a long time building those technical skills. And it turns out that…
Brian Teller
Yeah, I think you're right. When you were building DBT, what made practitioners trusted enough to put it into their production workflow? How did you make that happen? How did you build that trust with the community?
Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
The opening question was what changes when an AI agent, not a person, is the main reader of a company's data. Handy argues the value moves to well-modeled tables, a semantic layer and context. Dashboards stay only as a way to show people what the agent found.
Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News · 2026-11-16
Permalink →
Listen →
Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News
Around 02:31 into the episode
Tristan Handy
Thanks for having me.
Brian Teller
So starting off, I'm curious. We've spent years building data platforms around people looking at dashboards. What changes when an AI Agents becomes the thing consuming that data?
Tristan Handy
Oh, gosh. Yeah, we're just diving right in, aren't we? Maybe the first and most kind of obvious thing is that dashboards aren't the product anymore. Dashboards are a hack for the human visual cortex. They are a way to pipe lots of information into our brains very quickly and allow us to notice patterns. And that makes a ton of sense when our brains are the ones that are looking for the patterns and applying the intelligence. But it is becoming true that agents are replacing us as the primary form of intelligence that is like looking at our data and trying to find patterns and the meaning in those patterns. And they don't really want to consume data via images. And so dashboards go away.…
Brian Teller
And when you say gold, you mean you're talking like a medallion setup? Okay. Just so our listeners that may not understand the gold, Braun and Silver. Yeah.
Tristan Handy
Totally. And I, you know, I am happy. I understand that the world of the data practitioner is a little different than the world of the SRE or DevOps practitioner. So I'm happy to go as deep or not deep as relevant. But yeah, assume that all of your data comes into your data lake in a pretty raw…
Brian Teller
And for this data, are we looking at this differently for an agent than we would a person, like a practitioner that's actually analyzing this data? AI Agents looking at it the same way? Do we have to reason about it differently? Do we have to structure it differently?
Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
The host asked what reliability means for agents working with company data, where a query can run fine and still lead to a wrong decision. Handy says writing SQL is solved. The hard part is the agent understanding what was actually asked, and he gives benchmark numbers for it.
Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News · 2026-11-16
Permalink →
Listen →
Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News
Around 09:13 into the episode
Tristan Handy
Wait, is your question about reliability in the context of agents, or is it just data generally?
Brian Teller
On the agent side specifically, if you can answer that.
Tristan Handy
Yeah, because it's a somewhat hard question to answer when humans are in the loop, too. That's fair. But there are benchmarks when it comes to evaluating agents and their accuracy for data-related tasks. I think that the one that comes up most often is called Spider. I think there's one Spider2. And the benchmarks like this attempt to model the complexity of real-world enterprise data, which is very messy. It has a tremendous number of different entities modeled in it. Join pathways are not clear, et cetera, et cetera. And you can then ask questions of this data set and use an agent to go get you the answers to these questions. And you can experiment with how you ask the questions, what tools you give the agents, what kind of context you give them and measure the difference in the results on the benchmark. There are pretty stark differences in the performance you will see. If I could summarize the kind of general industry wisdom on these evals today, it is that agents are very good at writing SQL. They can write SQL that Compiles and executes against your database with almost 100% accuracy, the problem today is not text-to-sQL. The challenge today is: did the agent fully understand the question that you were asking? Did they understand the context of the data? And did they actually get you the…
Brian Teller
Yeah. No, it makes 100% sense. I guess kind of pulling on that thread, now talking about the governance side of it, if an agent can discover data and act on it, where do we put the boundary between useful autonomy and access that you'll end up regretting?
Tristan Handy
Say more about that. Where's the regret coming from?
Brian Teller
Well, I'm more thinking like the difference between reading data and taking action. And that taking action step can obviously devolve into some crazy modifications of your application, your infrastructure. So where do we draw that line?
Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
Handy was reacting to the guest's argument that AI-generated charts need to be auditable. He says humans don't have to review all code forever, but code has to stay reviewable if you want to, and that is why they designed a readable spec.
The Analytics Engineering Podcast · 2026-10-30
Permalink →
Listen →
The Analytics Engineering Podcast
Around 31:25 into the episode
Tristan Handy
…And it turned out that I was able to build an absolute garbage version of this in, I don't know, I think it was like two days or something, which was enough to make me think like, yeah, the ergonomics of this are really good, but there's really going to be work involved in making the thing useful.
Dave Fowler
…here, it's coming. And how do we make sure that people and AI can work well together? And having that right seam where like it's still this thing that you can audit and test and humans can still have an eye on it and monitor and govern that data, I think is going to be really, really important.
Tristan Handy
Having something that is agent writable and human reviewable is actually not a super easy design constraint to satisfy. But I think so, so important. I have definitely spoken to folks on data teams where they've gone so far down the rabbit hole of having agents write all of their code that they've lost the ability to review it. And that gets to this, you know, I'm not saying that humans necessarily need to review all code forever, but I think it's important for it to be reviewable if you want to review it. Because otherwise, the structure of the conversation of like, hey, are you sure that this is correct? Is just like I cannot tell you on my own authority that it's correct. I'm going to turn around to my agent and ask it that question. And I'm then going to have to rely on its answer. So I think that that design criteria is exactly the right one.
Dave Fowler
I think even in a future, I think about a future where like we're not even looking at the code, you know, it's still important to have those right level of abstractions, even for the AI. Like Markdown is inherently valuable to AI. It's like abbreviates like its stuff. It uses less tokens. It's a…
Tristan Handy
So as in everything else, a new technology paradigm changes the way that we think about constructing our tools. We talked about the transition to the cloud before and how Chartio and a bunch of other companies were, you know, responses to the rise of the cloud. And this is DBT charts is an answer…
Dave Fowler
You know, in hindsight, you know, in the early days of Chartio, like my big competitor was Looker. And, you know, I kind of disliked its whole philosophy. In hindsight, you know, a lot of the code first, they were really trying to do this code first. You could even do these, kind of do these…
Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
Talking about whether AI models will converge or diverge over time, Handy reaches for evolution. Benchmarks, he argues, act like the predators and food scarcity that shaped humans — the environment a model adapts to — so models pushed against the same benchmarks will tend to converge on the same behavior.
The Analytics Engineering Podcast · 2026-08-21
Permalink →
Listen →
The Analytics Engineering Podcast
Around 36:29 into the episode
Tristan Handy
for working that in.
Jason Ganz
…like, it's a bit of an open question to me on how much, how much we'll actually see that. And then therefore, how much true kind of fluidity in the at the highest layers. There's always going to be some set of workloads that are addressable within this. It's just like a question of what percentage.
Tristan Handy
…router represents. But I agree that's a long-term statement and it's not clear. Over time, I think that the evolutionary, we were talking about benchmarks right before this. And benchmarks are, you can think of them almost as like evolutionary pressure. How did humans evolve in environments where there were large predators or a scarcity of food or a scarcity of what? Like these are, you know, benchmarks become the environment that these models are evolving inside of. And to the extent that all the models are. Responding to the same evolutionary pressure, then they will converge on some time horizon. But the other case, I guess, is that you have agents that are very explicitly, or sorry, models that are very explicitly tuned to be better in some evolutionary conditions versus others. Maybe that trade-off makes sense. And there's like the class of model that…
Jason Ganz
That's right. And if you're a data analyst at Stripe and you want to come on and talk to Tristan Ley about tokenomics, let us know.
Tristan Handy
Where are we going from here?
Jason Ganz
Okay. So I think a perfect way to close this out is, you know, in the benchmarking section, we were talking about the fact that models have gotten pretty good at the set of the things that we are currently addressing within data orgs. And like the fundamentals are still very important and you still…
Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.