ST

Stephen Toub

Things Stephen Says on Podcasts

Where to Find Them

Stephen Toub writes GitHub Blog . They have also been a guest on The Developing Dev and The Peterman Pod .

Recently: “Microsoft Distinguished Eng: Why And How To Port Your Entire Codebase to Rust | Stephen Toub” on The Developing Dev (November 2026); “Microsoft Distinguished Eng: Why And How To Port Your Entire Codebase to Rust | Stephen Toub” on The Peterman Pod (November 2026); “Migrating the GitHub Copilot runtime to Rust, using Copilot” on GitHub Blog (September 2026).

What They Said

“It was about 136 billion tokens broken up in such a way that if you were to use the public pricing of those models at those times, it came out to approximately $120,000. This is not factoring any discounts… plus my time… about three weeks of my time spread out over three months plus the token costs. And it's only getting cheaper.” — Stephen Toub, The Developing Dev

Asked about the cost of rewriting the Copilot runtime in Rust with agents, Toub totals the token spend at public list prices and estimates his own time. It is a rare concrete price tag on an agent-driven migration of several hundred thousand lines.

The Developing Dev · 2026-11-02 Permalink → Listen →
The Developing Dev
Stephen Toub

…different paths and looking at different things. And one of the biggest changes you can make is completely changing the model. And so the best way to get different opinions and focusing on different things that I found is by having different models. Especially than the one that I used initially.

Ryan Peterman

So, you know, after you finish the rewrite, what was the concrete impact and the cost? So, if you look at it from a business perspective, what was the benefit and the cost of this project?

Stephen Toub

…previous world that I was talking about. So, from a business perspective, it's worth a lot to do this port. In terms of what it actually costs us, it was about if I, so I stored all of my session logs or 99% of my session logs from this effort. And from those session logs, I was then able to sort of deduce the token counts for fresh tokens, cash writes, cash reads, output tokens. And all told, it was about 136 billion tokens broken up in such a way that if you were to use the public pricing of those models at those times, it came out to approximately $120,000. This is not factoring any discounts or things that companies might have for what those tokens cost. And this is just sort of what's on the website of OpenAI and Anthropic, plus my time, which it took, it was about 14, 15 weeks, but I was doing other things at the same time. A lot of these agentic projects, if I look at my co-pilot app that I currently have on my screen, I've got about 35 sessions right now for things that I'm…

Ryan Peterman

Wow. So three weeks of, I guess, of developers' time plus $120,000, you're able to rewrite the entire code base. If you were to scope this out pre-AI, let's say you wanted to undertake this, what would that look like?

Stephen Toub

First, it wouldn't have happened. And I'm actually serious, like for this particular thing, it wouldn't have happened because by the time we were done, we would have ported the wrong thing. This space is moving so fast that we would have basically never finished, I think. In fact, you can kind of…

Ryan Peterman

It sounds like, at least for this type of work where it's super what we want is very obvious and encoded somewhere in the existing functionality. So like this type of work is at least one order of magnitude cheaper, maybe even more. I mean, if you had a team of engineers and they're, you know…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
“The agents want to get to green, like that's their goal. And so if you let them, they will change the oracle, like they will change the tests at the same time as the code to make it pass.” — Stephen Toub, The Developing Dev

Asked how regressions were prevented while porting GitHub Copilot's runtime from TypeScript to Rust with agents, Toub says end-to-end tests were the main safeguard. One failure was not stopping agents from editing those tests while changing the behavior they covered.

The Developing Dev · 2026-11-02 Permalink → Listen →
The Developing Dev
Stephen Toub

…find their way into the existing untouched TypeScript in the branch. And then the agent would have to reason about all of the code that it had ported, looking for changes to that to bring that in, versus it was sort of instantly pointed to those places because we were doing this swap at the time.

Ryan Peterman

…or engineers that want to rewrite their entire code base. Probably the biggest concern is the correctness concerns, which is there's so much code that you don't have the luxury of a human even reading the code to confirm that it's correct. So how did you discover and fix regressions along the way?

Stephen Toub

…good as long as you have tests covering all the behaviors you care about. And so the fact that we had any regressions sort of highlights that there were gaps. Now, that's not 100% true. We had cases, the funniest or saddest cases were where we neglected to prevent the agents from rewriting the end-to-end tests at the same time that they were changing the behavior that was being tested. You know, the agents want to get to green, like that's their goal. And so if you let them, they will change the oracle, like they will change the tests at the same time as the code to make it pass. Or we had one case where this one didn't slip through. I noticed this one, but like the agent was making changes and it ended up changing some public APIs. And CI caught this and said, no, no, no, you can't do that. But we have an escape hatch in CI where if someone really knows what they're doing, they can put a label on the PR to basically say, no, this particular break is okay. And the agent…

Ryan Peterman

It still feels like a regression is inevitable because the test coverage is, it's like it's opt-in, kind of where it relies on a human saying these are the things that are important, but you could always forget something. So I kind of wonder if this type of project, it's just inevitable that there…

Stephen Toub

I think so. I mean, it's hard to imagine. I mean, who knows what things will look like in a year. Things are moving really fast. But right now, yes, 100% there will be regressions. If there aren't, I would be astounded. Unless we're talking about like, you know, a tiny, tiny little code base. But…

Ryan Peterman

For such a huge volume of code, I imagine. Imagine most of it wasn't human reviewed. How did you handle code review at scale, or did you do any at all?

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

Collections They Appear In

Software Engineering & Infra