“You know, the agents want to get to green. That's their goal. And so if you let them, they will change the oracle. They will change the tests at the same time as the code to make it pass.”
The Peterman Post · Software Engineering & Infra · November 2026
Context
Asked how he caught regressions in a port too big to read by hand, Toub says end-to-end tests were the safeguard, and describes the main way agents undermine them.
Transcript
…find their way into the existing untouched TypeScript in the branch. And then the agent would have to reason about all of the code that it had ported, looking for changes to that to bring that in, versus it was sort of instantly pointed to those places because we were doing this swap at the time.
…or engineers that want to rewrite their entire code base. Probably the biggest concern is the correctness concerns, which is there's so much code that you don't have the luxury of a human even reading the code to confirm that it's correct. So how did you discover and fix regressions along the way?
So the most important thing was our end-to-end tests, our integration tests, or whatever you want to call them. We basically had two flavors of tests for Copilot. We had unit tests, which were largely sort of, you know, they're testing micro functionality throughout the code base, and a lot of them have internals access. So they're able to test this little function here that's not exposed or that piece of code there that's not exposed. And then there are the end-to-end tests either at the runtime level, the SDK level, or the CLI is in the same code base. And so its end-to-end tests also effectively exercise the runtime. And this isn't 100% true, but the CLI can largely be thought of as sort of just exposing the runtime's capabilities and sort of the lowest layer possible in a human interface. And so we had all of those end-to-end tests to validate behaviors. And by and large, if not 100%, but the vast majority of regressions that we had were because we didn't have…
It still feels like a regression is inevitable because the test coverage is, it's like it's opt-in, kind of where it relies on a human saying these are the things that are important, but you could always forget something. So I kind of wonder if this type of project, it's just inevitable that there…
I think so. I mean, it's hard to imagine. I mean, who knows what things will look like in a year. Things are moving really fast. But right now, yes, 100% there will be regressions. If there aren't, I would be astounded. Unless we're talking about like, you know, a tiny, tiny little code base. But…
For such a huge volume of code, I imagine. Imagine most of it wasn't human reviewed. How did you handle code review at scale, or did you do any at all?
More from The Peterman Post
