“A team that's achieving 90% isn't really honest about where we are doing well and where we're not. I would much rather see 50% and understand exactly where we need to focus, rather than 90% and we're celebrating. You'll never get to 100, right? It's statistical. If you got to 100, something's up.”

…would fight for like percentages, you know, like one, two, three percent was like a big gain and you're kind of like very painfully hill climbing. Now we're starting to see like I can saturate an eval or performance in like months versus what it used to be where it was like a year and we pushed 5%.

So when you tell them to redo their evals, what does that mean? Do they scrap it, start over? Do they have a base to start hill climbing on?

So a lot of evals are sort of looking for data that hasn't been trained into the model to figure out how to get an eval that is novel responses, right? Do you look for that? Or like you said, would you be happier if it was just like there was a lower level where you kicked it to a human?

Well, I think in our case, because we are what I love about this place is quality, quality, quality. So I think we're always going to focus on making sure that the end user has the best experience, right? We're not deploying these systems for the sake of like maximalizing AI. We're trying to deploy…

Yeah, I want to get introspective on Airbnb. I know this is the sort of external use case for AI, but a lot of companies are bringing AI inside and getting agentic SDLCs. And I'm curious, I was at a conference last week and somebody said, agentic AI coding tools are really accelerating individual…