André Meyer had just said quality is one of the hardest things to measure, and named rollbacks and post-release failures as useful signals. Brian Houck adds what he says DX's data across many organizations shows about AI and change failure rate.
I'm curious on that. Is there a dimension that you think gets missed most often?
…very missed a lot and the collaboration aspects. But also I think quality remains very hard to measure and the impact on quality. But I think there are a couple of metrics that are helpful in that regard, such as rolling back releases afterwards and failures that are found after releases and so on.
Yeah, I mean, change failure rate is something that I'm really interested in looking at those patterns relative to AI adoption. Proving the point on why looking across dimensions is so necessary. We see across hundreds of organizations, change failure rate doesn't seem to be getting better as we're using AI more. And like, that's something we need to keep our eyes on. So, shifting gears a little bit, as I mentioned in the intro, when we first met, you were working on your good day study. Could you just explain very briefly to the audience what that study was, what the questions you were trying to answer?
So, this was a project that I've done in one of my internships at Microsoft Research together with Tom Simmerman and Chris Bird and Earl Barr as well. He was a visiting professor at the time. And that's also where I've met you. And we wanted to better understand what makes developers' workdays good…
And unfortunately, what we're seeing is this tension now with AI between developer sentiment and sort of the amount of work we're producing. Annie Bella calls it the productivity experience paradox. And I find that just like a very interesting phenomenon that is terrifying in many ways. So you were…
Yeah, it's so much fun to reflect back on the work that I've done initially when I started my research in the space. And the very first study that I've done were those observational studies. And it was super weird, actually, because I had to sit behind developers for several hours. I had to sit…