SA Sabina Anja On Software Engineering Radio - the podcast for professional software developers

“In the old world, we actually never quite saturated the network. … What we see with AI traffic is that collectives do send all the traffic in full send all the time. So if you have an oversubscribed fabric, every sync just runs at full link speed. … Traffic shape in AI, and in general for collectives, is basically one pipe that's full. You're taking advantage of everything all the time.”

Software Engineering Radio - the podcast for professional software developers · Hands-On Engineering Podcasts · November 2026

“In the old world, we actually never quite saturated the network. … What we see with AI traffic is that collectives do send all the traffic in full send all the time. So if you have an oversubscribed fabric, every sync just runs at full link speed. … Traffic shape in AI, and in general for collectives, is basically one pipe that's full. You're taking advantage of everything all the time.” — Sabina Anja, Software Engineering Radio - the podcast for professional software developers

Context

Sabina explains why oversubscription, a standard way to save money in enterprise networks, breaks down in AI clusters. Traditional traffic was bursty and rarely used every port at once, while GPU collectives saturate every link together.

Transcript

Software Engineering Radio - the podcast for professional software developers Around 20:29 into the episode
Sabina Anja

…that's more of a scale-out system. So, we've gone out of the server at this point, and we can have collectives and collective libraries that move traffic between these same-ranked GPUs. That way, the topology matches the traffic. This, by the way, is NVIDIA's reference design for the H100 clusters.

Amey Ambade

So, can you tell me about the intra-node versus inter-node distinction within NV-Link, for example, or using something like InfiniBad or Ethernet between boxes?

Sabina Anja

Whether you use InfiniBand or Ethernet between boxes with RDMA and so on, that's entirely up to you. Like, that's a technological question, and I think you're fine regardless, right? Within the server, which is your first question, is kind of what we talked about now, right? How do you design for the best GPU cluster and how do you leverage all of that insight? The question that you're asking on the InfiniBand on RDMA over Ethernet, which is your other type of technology, is more about how do you design for the network to connect between the different servers, again, in the best possible way. And that Rail optimized design is one solution. The other one is a dedicated network just for your back-end AI traffic. So, it's not the first time we've done this. In the past, we used to have something called a SAN network, which was a dedicated network just for storage. And this comes from the days of you needed lossless, you needed low latency, you needed no interference on your storage nodes, because if you'd have that, you'd potentially lose a backup. You'd potentially lose data that you needed to move between place A and place B. So, what happened for a long time is: hey…

Amey Ambade

So, let's zoom out of the room. As you said, data center networks are traditionally designed, assuming that most of the traffic kind of goes in and out of the building, and AI clusters flip that, where most of the traffic is GPUs talking to each other side by side.

Sabina Anja

Yep.

Amey Ambade

Can you talk more about how that actually changes the layout of the switches and the links in here?

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from Software Engineering Radio - the podcast for professional software developers