“If they one-shotted an entire feature, I'm naturally going to be skeptical. Sometimes I'm skeptical and then surprised positively that it did a great job. Sometimes you use a very long session to solve something that seems like it should be a one-shot, and it maybe tells you something about why your code base is hard to reason about. Your agents are having trouble reasoning around your code base in some way.”
devtools.fm: Developer Tools, Open Source, Software Development · Hands-On Engineering Podcasts · November 2026
Context
A host said it's hard to tell a one-shot prompt from a session someone spent an hour on, and a trace shows the difference. Tarun adds that comparing task difficulty with session length can reveal problems in a codebase, not just in how people work.
Transcript
…those are the two things that I think are the most important to us and where we've seen the most value. But yeah, it's not quite a perfect marriage of those two tools quite yet. I think there's a lot more points of integration there. I'm not sure if that fully answers your question, but like, yeah.
…on this? Like, how many tokens are spent? Like, how long was this actual session? Like, you know, it's a very different story between like, okay, I'm looking at the trace and there was like one prompt at the very top and then you know, like 30 minutes of the agent spinning. Yeah, exactly. Exactly.
It's like, yeah, I mean, and sometimes that works. Sometimes that works really, really well. It's tough to make. I haven't seen too many instances of, you know, I guess over time, this is like longer and longer horizon tasks are becoming more and more feasible. It's, you know, it's, it's, I would say it's hard to kind of like, and at least this is not something we do internally. We don't like track how many turns necessarily takes to solve how many types of features and stuff like that. But those metrics in aggregate over time, you probably will want to have some visibility into. And like, you know, we can surface that information, but, you know, knowing how someone arrived at something helps me understand what level of thought and depth they put into that. Andrew Lisowski that appropriate for the task, right? Like, if they one-shotted an entire feature, like I'm naturally going to be skeptical. Sometimes I'm skeptical and then surprised positively that actually it did a great job, right? And there's something to be learned from how that one-shot worked and why it worked. Sometimes you use a very long session to solve something that seems like it should be a one-shot, and it maybe tells you…
yeah, I think like a, I don't know, like a, there's a surface here, you know, about like effort, effort spent, or like, I don't know, something.
Yeah, yeah, exactly. Exactly. We do have that some of that cost data in tokens in like our token surface, which gives you like turn-by-turn view on like how much every turn cost you. And, you know, reveals a lot of your, you know, things around like model choices and things like that. But yeah…
yeah, the coding harness I use at work recently added a dollar amount to each like session. And seeing that dollar count go up as I like iterate more and more, I'm like, oh, this will just be like a nice little feature I can fire off. And it was like $400 later. Sheesh.
More from devtools.fm: Developer Tools, Open Source, Software Development
