SA

Sabina Anja

Things Sabina Says on Podcasts

Where to Find Them

Sabina Anja has been a guest on Software Engineering Radio - the podcast for professional software developers .

Recently: “SE Radio 747: Sabina Anja on How AI Workloads Shape GPU Network Design” on Software Engineering Radio - the podcast for professional software developers (November 2026).

What They Said

“In the old world, we actually never quite saturated the network. … What we see with AI traffic is that collectives do send all the traffic in full send all the time. So if you have an oversubscribed fabric, every sync just runs at full link speed. … Traffic shape in AI, and in general for collectives, is basically one pipe that's full. You're taking advantage of everything all the time.” — Sabina Anja, Software Engineering Radio - the podcast for professional software developers

Sabina explains why oversubscription, a standard way to save money in enterprise networks, breaks down in AI clusters. Traditional traffic was bursty and rarely used every port at once, while GPU collectives saturate every link together.

Software Engineering Radio - the podcast for professional software developers · 2026-11-20 Permalink → Listen →
Software Engineering Radio - the podcast for professional software developers Around 20:29 into the episode
Sabina Anja

…that's more of a scale-out system. So, we've gone out of the server at this point, and we can have collectives and collective libraries that move traffic between these same-ranked GPUs. That way, the topology matches the traffic. This, by the way, is NVIDIA's reference design for the H100 clusters.

Amey Ambade

So, can you tell me about the intra-node versus inter-node distinction within NV-Link, for example, or using something like InfiniBad or Ethernet between boxes?

Sabina Anja

Whether you use InfiniBand or Ethernet between boxes with RDMA and so on, that's entirely up to you. Like, that's a technological question, and I think you're fine regardless, right? Within the server, which is your first question, is kind of what we talked about now, right? How do you design for the best GPU cluster and how do you leverage all of that insight? The question that you're asking on the InfiniBand on RDMA over Ethernet, which is your other type of technology, is more about how do you design for the network to connect between the different servers, again, in the best possible way. And that Rail optimized design is one solution. The other one is a dedicated network just for your back-end AI traffic. So, it's not the first time we've done this. In the past, we used to have something called a SAN network, which was a dedicated network just for storage. And this comes from the days of you needed lossless, you needed low latency, you needed no interference on your storage nodes, because if you'd have that, you'd potentially lose a backup. You'd potentially lose data that you needed to move between place A and place B. So, what happened for a long time is: hey…

Amey Ambade

So, let's zoom out of the room. As you said, data center networks are traditionally designed, assuming that most of the traffic kind of goes in and out of the building, and AI clusters flip that, where most of the traffic is GPUs talking to each other side by side.

Sabina Anja

Yep.

Amey Ambade

Can you talk more about how that actually changes the layout of the switches and the links in here?

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
“The model for inference is it generates one token at a time. So if your model is sharded across GPUs, those GPUs synchronize on each and every token. You're paying your network latency once per token. So it's 100 times if you have a long answer.” — Sabina Anja, Software Engineering Radio - the podcast for professional software developers

Asked how network behavior at inference differs from training, Sabina explains that a model split across GPUs pays network latency on every generated token. That makes time to first token a useful check on infrastructure health.

Software Engineering Radio - the podcast for professional software developers · 2026-11-20 Permalink → Listen →
Software Engineering Radio - the podcast for professional software developers Around 39:57 into the episode
Sabina Anja

…Or is it the problem with the NIC? Like where is exactly the issue? But when you have this kind of like slightly more systems issue, you actually get a little bit more input. So it's slightly easier to narrow it down, but it also means kind of opening up that ticket with your infrastructure team.

Amey Ambade

…assuming that there's something there that's just giving me an answer right immediately. If the model is too big for one GPU, serving it, I'm guessing that also becomes a networked collective kind of problem, right? So how is the network behavior at inference time different than during training?

Sabina Anja

So the thing is, inference actually has a different shape, right? The model for inference is it generates one token at a time, right? So if your model is sharded across GPUs, those GPUs synchronize on each and every token. Now, back to your network latency question. You're paying your network latency once per token. So it's 100 times if you have a long answer. Tokenomics and especially time to first token, remember, is a very good measure of infrastructure performance because the time it takes you to get a token, and especially if you can multiply that by the amount of tokens you're spreading throughout the network, give you a very good sense of how well your infrastructure is built, behaving, and set up for AI. And a lot of observability tools focus on giving you that number as soon as you have that in your infrastructure. And the way users feel that is, you know, that I'm sure you had this with days when OpenAI or Anthropic or whatever are not having one of the best days. You're seeing the text coming like one, two, three…

Amey Ambade

In bursts.

Sabina Anja

Exactly. Slow bursts just to get the frustration out of there. And it's usually right before a deadline when you're thinking, like, you know, I have five minutes to finish this. Come on, Claude, just get on with it. There's a key thing on inference, right? Because it's not just this, it's the fact…

Amey Ambade

So what I understand is that there's a useful split in inference between the phase that reads the whole prompt and the phase that generates these tokens one by one and that they have different appetites. One can be compute heavy, one can be more memory and latency bound.

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

Collections They Appear In