Software Engineering Radio - the podcast for professional software developers · Hands-On Engineering Podcasts · November 2026
Asked how network behavior at inference differs from training, Sabina explains that a model split across GPUs pays network latency on every generated token. That makes time to first token a useful check on infrastructure health.
…Or is it the problem with the NIC? Like where is exactly the issue? But when you have this kind of like slightly more systems issue, you actually get a little bit more input. So it's slightly easier to narrow it down, but it also means kind of opening up that ticket with your infrastructure team.
…assuming that there's something there that's just giving me an answer right immediately. If the model is too big for one GPU, serving it, I'm guessing that also becomes a networked collective kind of problem, right? So how is the network behavior at inference time different than during training?
So the thing is, inference actually has a different shape, right? The model for inference is it generates one token at a time, right? So if your model is sharded across GPUs, those GPUs synchronize on each and every token. Now, back to your network latency question. You're paying your network latency once per token. So it's 100 times if you have a long answer. Tokenomics and especially time to first token, remember, is a very good measure of infrastructure performance because the time it takes you to get a token, and especially if you can multiply that by the amount of tokens you're spreading throughout the network, give you a very good sense of how well your infrastructure is built, behaving, and set up for AI. And a lot of observability tools focus on giving you that number as soon as you have that in your infrastructure. And the way users feel that is, you know, that I'm sure you had this with days when OpenAI or Anthropic or whatever are not having one of the best days. You're seeing the text coming like one, two, three…
In bursts.
Exactly. Slow bursts just to get the frustration out of there. And it's usually right before a deadline when you're thinking, like, you know, I have five minutes to finish this. Come on, Claude, just get on with it. There's a key thing on inference, right? Because it's not just this, it's the fact…
So what I understand is that there's a useful split in inference between the phase that reads the whole prompt and the phase that generates these tokens one by one and that they have different appetites. One can be compute heavy, one can be more memory and latency bound.