Things Ahmad Says on Podcasts

…of deploying these tools at scale. And I think it's actually counterintuitive, but it's a complicated situation. The senior engineers have judgment. They've shipped things, they've written artisanal code, they've handwritten code, they've deployed it into production, they've been burned. They

broke prod several times, right?

You know, with a lot of folks I talk to are building software factories, right? To go with the factory analogy, they're building plow shares and not swords. But the judgment has to be engineered into the system pretty sort of explicitly, right? And I think codifying that sort of ties and knowledge…

We've been experimenting with a few things that are interesting, but I don't think are like cures. You know, we have this like concept of context graph that encodes institutional knowledge. It's just like a shared memory bank, like a company brain, if you will, which we're hoping to like be able to…

Another issue I've seen, you know, in a lot of larger engineering organizations, you lock down the tools, the languages, the things that any developer can use, right? With AI, it seems like I've talked to folks who their organization just has thousands of skills, many that do the same thing, little…

…commiserate with the speed that AI gives you. Still improving. We still have a lot more work to do, but that's kind of the mentality you want to bring is like, are we moving at the pace that we should be? And is our human tax low enough that the cost is mostly associated to building and iterating?

I think there's a concern with the increased speed of delivering code to production that there's a reduction in quality too. So I wonder if you notice any sort of change in incidence, production outages.

To go back to the productivity question, are things like Dora metrics still optical or do we need new ways to measure the velocity of a development team?

The process of building product and features and deploying things into production is an inherently multi-disciplinary problem. So in addition to like rethinking how software engineering velocity works, you kind of have to think about the whole software development lifecycle. And that's where I…

Are there many parts of a piece of business critical software that are just committed and quit it, like doesn't need anybody but an engineer on it?

…would fight for like percentages, you know, like one, two, three percent was like a big gain and you're kind of like very painfully hill climbing. Now we're starting to see like I can saturate an eval or performance in like months versus what it used to be where it was like a year and we pushed 5%.

So when you tell them to redo their evals, what does that mean? Do they scrap it, start over? Do they have a base to start hill climbing on?

So a lot of evals are sort of looking for data that hasn't been trained into the model to figure out how to get an eval that is novel responses, right? Do you look for that? Or like you said, would you be happier if it was just like there was a lower level where you kicked it to a human?

Well, I think in our case, because we are what I love about this place is quality, quality, quality. So I think we're always going to focus on making sure that the end user has the best experience, right? We're not deploying these systems for the sake of like maximalizing AI. We're trying to deploy…

Yeah, I want to get introspective on Airbnb. I know this is the sort of external use case for AI, but a lot of companies are bringing AI inside and getting agentic SDLCs. And I'm curious, I was at a conference last week and somebody said, agentic AI coding tools are really accelerating individual…