Software Engineering Radio - the podcast for professional software developers · Hands-On Engineering Podcasts · November 2026
Sabina explains why oversubscription, a standard way to save money in enterprise networks, breaks down in AI clusters. Traditional traffic was bursty and rarely used every port at once, while GPU collectives saturate every link together.
…that's more of a scale-out system. So, we've gone out of the server at this point, and we can have collectives and collective libraries that move traffic between these same-ranked GPUs. That way, the topology matches the traffic. This, by the way, is NVIDIA's reference design for the H100 clusters.
So, can you tell me about the intra-node versus inter-node distinction within NV-Link, for example, or using something like InfiniBad or Ethernet between boxes?
Whether you use InfiniBand or Ethernet between boxes with RDMA and so on, that's entirely up to you. Like, that's a technological question, and I think you're fine regardless, right? Within the server, which is your first question, is kind of what we talked about now, right? How do you design for the best GPU cluster and how do you leverage all of that insight? The question that you're asking on the InfiniBand on RDMA over Ethernet, which is your other type of technology, is more about how do you design for the network to connect between the different servers, again, in the best possible way. And that Rail optimized design is one solution. The other one is a dedicated network just for your back-end AI traffic. So, it's not the first time we've done this. In the past, we used to have something called a SAN network, which was a dedicated network just for storage. And this comes from the days of you needed lossless, you needed low latency, you needed no interference on your storage nodes, because if you'd have that, you'd potentially lose a backup. You'd potentially lose data that you needed to move between place A and place B. So, what happened for a long time is: hey…
So, let's zoom out of the room. As you said, data center networks are traditionally designed, assuming that most of the traffic kind of goes in and out of the building, and AI clusters flip that, where most of the traffic is GPUs talking to each other side by side.
Yep.
Can you talk more about how that actually changes the layout of the switches and the links in here?