Gmyr runs adversarial AI reviews on his own projects. Each round finds one more edge case, the fix invites another round, and a small change ends up handling 20 cases nobody asked for. He and TJ Miller are still working out where to draw the line.
…CodeRabbit before for personal stuff. I'll have to look at it again because I do the adversarial reviews too. And also I'm trying to build my own set of review skills for my personal projects. But it is so hard to find the balance of like, yes, you found something, but is this actually important?
Right.
…a valid, you know, call out. Let me fix that. And then the next thing, it bypasses the thing that you just fixed, but now it finds issue number two. Oh, yeah, issue two makes sense. We should fix that. And then now this like little itty bitty change has exploded into five, 10 different commits and it's still going and still being adversarial and finding these like little nitpicky things. And your scope has blown up 10x, your PR has blown up 10x. And now it's just like my one, you know, little change to a component or even, you know, a design or whatever the case is. It doesn't even matter. It's like it's blown completely out of proportion and handles all these like 20 different edge cases. And it's like, why? Why can't we just ship it? And then if an issue comes up, then we'll fix it then. And, you know, some things like it does find…
Yeah, I was literally just wrestling with that yesterday of like, it just kept on finding things no matter what. And it's like, how do I, how do I prompt its way into, you know, where that line even is? And like, how do you define that? And geez, like, it's, it's, it feels really nuanced. And cause…
that. Yep, exactly. And the other thing that's been on my mind is like how to test and tune these things. So with like my AI review script bot, whatever, I do a lot of like manual testing and I'll run like the dev branch on like repos that I own. So I can be like, you know, how does this feel…
I, Claude and I built an eval harness as part of the work because that's, that's just like every AI feature that I have built at Luma for the platform. I build eval harnesses to go along with them so so that Claude can iterate on its own and kind of like run the eval, tweak some things, run the…