“They were designed for a world where producing code was really costly and now it's almost free. It's super cheap. And so those metrics inflate a lot and they're also very easy to game. And shipping more doesn't really mean that you're creating anything of value.”
Engineering Enablement · Software Engineering & Infra · November 2026
Context
Brian Houck asked why it is so hard to answer "we've invested in AI, but has it helped?" André Meyer said most dashboards track AI use and output, like adoption, acceptance rates and pull request counts. This is his explanation of why classic output metrics stop meaning much once AI writes the code.
Transcript
Hi, Brian. Thanks so much for having me. It's always super exciting to talk to you about good developer experience.
…leader being asked, you know, we've invested in AI, but has it helped? And leaders today frequently have dashboards, but they rarely have clear answers from those dashboards. Like, why is it such a difficult question to answer? Like, you know, has AI helped? Like, has our investment in AI helped?
Yeah, from talking to developers and building the data set around the COMPASS, which by the way, includes metrics from around 40 companies, we've noticed that most dashboards are only covering AI use and AI output. So instead of actual outcomes and value delivered. So they would look at metrics around AI adoption, daily active users, acceptance rates, in some cases, even lines of code or token spend. And those are really easy to collect, which is why they are so commonly used. But it only shows if AI is being used and it doesn't really show how it helps to create more value or even true return on investments. And also we've noticed that they use a lot of sort of classic output-related measures like throughput and number of pull requests and deployment frequency and things like that. And they were already a challenge before AI to like compare between teams and divisions and so on, but this got even more ambiguous with AI. The problem is that they were designed for a world where producing code was really costly and now it's almost free. It's super…
So that's really interesting, sort of the idea that whether or not AI is helping is a difficult question. to answer because the easiest metrics to measure are the ones that tell us sort of the least about outcomes. And to try to help us reason about measures better, in your paper, you mapped out…
So as you've mentioned, we've looked into all the measures that we could find in the engineering space, so what companies are actually applying, but also from the research space. And we've seen that there are a lot of metrics and we started to collect them in a systematic way and we've gathered the…
For those who have not seen the DevEx Metrics Compass, I like cannot encourage you enough. It is beautiful and just like awe-inspiring how many different metrics are sort of mapped together to different frameworks. And it's like, it's a lot of fun to go and dive into. And so as you were creating…
More from Engineering Enablement
