“The agents want to get to green, like that's their goal. And so if you let them, they will change the oracle, like they will change the tests at the same time as the code to make it pass.”
The Developing Dev · Software Engineering & Infra · November 2026
Context
Asked how regressions were prevented while porting GitHub Copilot's runtime from TypeScript to Rust with agents, Toub says end-to-end tests were the main safeguard. One failure was not stopping agents from editing those tests while changing the behavior they covered.
Transcript
…find their way into the existing untouched TypeScript in the branch. And then the agent would have to reason about all of the code that it had ported, looking for changes to that to bring that in, versus it was sort of instantly pointed to those places because we were doing this swap at the time.
…or engineers that want to rewrite their entire code base. Probably the biggest concern is the correctness concerns, which is there's so much code that you don't have the luxury of a human even reading the code to confirm that it's correct. So how did you discover and fix regressions along the way?
…good as long as you have tests covering all the behaviors you care about. And so the fact that we had any regressions sort of highlights that there were gaps. Now, that's not 100% true. We had cases, the funniest or saddest cases were where we neglected to prevent the agents from rewriting the end-to-end tests at the same time that they were changing the behavior that was being tested. You know, the agents want to get to green, like that's their goal. And so if you let them, they will change the oracle, like they will change the tests at the same time as the code to make it pass. Or we had one case where this one didn't slip through. I noticed this one, but like the agent was making changes and it ended up changing some public APIs. And CI caught this and said, no, no, no, you can't do that. But we have an escape hatch in CI where if someone really knows what they're doing, they can put a label on the PR to basically say, no, this particular break is okay. And the agent…
It still feels like a regression is inevitable because the test coverage is, it's like it's opt-in, kind of where it relies on a human saying these are the things that are important, but you could always forget something. So I kind of wonder if this type of project, it's just inevitable that there…
I think so. I mean, it's hard to imagine. I mean, who knows what things will look like in a year. Things are moving really fast. But right now, yes, 100% there will be regressions. If there aren't, I would be astounded. Unless we're talking about like, you know, a tiny, tiny little code base. But…
For such a huge volume of code, I imagine. Imagine most of it wasn't human reviewed. How did you handle code review at scale, or did you do any at all?
More from The Developing Dev
