The commonest waste in a GPU cluster is not failing to get cards; it is the network falling behind: cards wait for data, switches queue, and overall throughput is pinned by one uplink. This guide does not cover model recommendations; it covers the judgement order — ask scale first, then interconnect, and only then discuss cables and modules.

We do not quote stock, and we do not assume you must buy a certain brand. Vendor architectures named below are public reference material, not deliverables of this company.

Why size the network before the cards

Inside one server, interconnect runs over NVLink／NVSwitch, bandwidth measured in TB/s; between machines it goes over the network, bandwidth measured in 100G／200G／400G. The two differ by an order of magnitude.

The result: the larger the cluster, the higher the share of cross-machine traffic, and the more the network becomes the bottleneck. A configuration that runs fast on one machine may slow down on east-west traffic once it joins eight or more machines. Before choosing cards, confirm how many machines, how much data must be exchanged between each pair, and what latency is acceptable.

Three scale thresholds

Scale Typical case Network focus Common mistake
1–2 machines Development, fine-tuning, single-machine inference In-box interconnect enough; general Ethernet outward Buying a high-end switch for a single machine that never uses it
8–32 machines Small-cluster training, shared by several users East-west traffic clear; needs low-latency fabric Only upgrading servers, switches still 10G／25G
64+ machines Large-model training, multi-tenant Fabric design, congestion control, predictability Using stacked access as backbone

The threshold is not an absolute number; it is the point where “cross-machine communication starts to dominate”. Past it, network design must be treated as a first-class engineering task, not a server accessory.

Interconnect options, and where they differ

Option Use Strength Cost／note
NVLink／NVSwitch In-box (scale-up) Highest bandwidth, lowest latency Limited by chassis and vendor ecosystem; not for cross-machine
InfiniBand Cross-machine fabric (scale-out) Low latency, mature congestion management, predictable behaviour Needs dedicated switches and tools; a current generation can be referenced in NVIDIA Quantum-X800 public architecture documents
RoCEv2 (RDMA over Converged Ethernet) Cross-machine fabric over Ethernet Reuses existing Ethernet skills and gear; flexible cost structure Needs lossless tuning; poorly tuned it performs worse than plain Ethernet
Ethernet fabric (e.g. NVIDIA Spectrum-X) Cross-machine Ethernet network Extends the Ethernet ecosystem, unified management Vendor public architecture; features and licences must be verified against vendor documentation
General Ethernet (TCP) Management network, storage, outward Lowest cost, most universal Higher latency and CPU overhead; not suited to large-scale training

In practice many clusters are mixed: training over IB or RoCEv2, management and storage over general Ethernet, the two networks kept separate. The benefit of separation is that backup tasks do not crowd out training traffic.

Three things to tune on RoCEv2

The RoCEv2 threshold is not buying gear; it is tuning. If these three are not done well, performance is worse than plain Ethernet:

  1. PFC (Priority Flow Control): lets a given priority not drop packets under congestion. Settings must match on both ends; one wrong port can trigger a chain pause along the whole path.
  2. ECN (Explicit Congestion Notification): signals congestion by marking rather than dropping packets. A threshold set too wide does nothing; set too narrow, throughput drops.
  3. Queue and buffer allocation: put RDMA traffic in an independent queue, avoiding contention with general traffic for the same buffer.

After tuning, verify — tuning is not done once and left: test with an actual training task or traffic generator, watch for PFC storms and excessive retransmission. Tuning without measurement equals no tuning.

100G／200G／400G: choosing optics and cables

Look at distance and environment first, then speed:

  • Same rack, under 3 m: DAC direct copper cable saves most; low power, low latency. Downside: thick, hard to cable, fails once distance grows.
  • Between racks, under 10 m: AOC active optical cable suits better; thinner than DAC, simpler than optics plus fibre.
  • Cross-row, cross-room, farther: optics ＋ fibre. Here module type matters (multimode／single-mode, reach, interface package).

The package per speed is largely standardised (100G commonly QSFP28, 400G commonly QSFP-DD／OSFP), but within one speed there are still reach grades and compatibility differences. Align three things: switch port spec, fibre type, actual distance. Any one mismatched, the link will not come up.

400G and 800G: when 800G is worth considering

800G-related speeds are specified in IEEE Std 802.3df-2024; from market observation, 400G still appears commonly mixed in existing and new deployments — this is not a stock statement. 800G generally becomes worth considering when GPU node density is high, east-west traffic is already approaching the 400G ceiling, and the switch and fibre support are in place; a new cluster should not default to 800G.

QSFP-DD vs OSFP: what a buyer should separate

  • QSFP-DD: keeps the QSFP form factor; oriented to density and existing chassis compatibility.
  • OSFP: larger body; thermal and power headroom is usually wider, so it is more often used for high-power modules.

Align port form factor, module power and rack cooling; matching speed alone does not mean the mechanical and thermal conditions work.

Compatible modules can be used, but “compatible” must be written into the procurement condition: which platforms, which firmware version, how replacement works on failure. Detail in the compatible optics selection guide and the compatible optics procurement page.

Cabling and racks: four easily missed details

  1. Polarity: MPO／MTP multi-core fibre has polarity types; wrong termination at either end brings the whole link down.
  2. Airflow direction: switches come in front-to-back and back-to-front; this must match the rack’s hot／cold aisle. Mixing causes some gear to overheat.
  3. Bend radius: high-density fibre bundles are easily over-bent in cable management; not visible short term, but attenuation over time.
  4. Labels and records: the port mapping table must be on paper. On a 64+ machine cluster, no port table means re-diagnosing every fault each time.

Capacity planning: measure east-west traffic first

The most valuable step before procurement is measuring, not guessing: peak east-west traffic (patterns differ between training-task start and checkpoint periods), the access-to-backbone oversubscription ratio, and storage traffic contending with training for the same path. For a new cluster with no historical data, estimate an upper bound as “network port count per machine × speed × expected convergence ratio”, then leave headroom.

Common misjudgments and used-gear checks

  1. Thinking swapping cards makes it faster: cards upgraded, switch still old speed; overall does not improve.
  2. Using stacked switches as backbone: stacking suits access, not a fabric backbone.
  3. Treating RoCEv2 as plain Ethernet: no lossless tuning, it ends up slower than before.
  4. Looking only at port count, not buffers: high-density small-buffer switches drop easily under burst traffic.

Used 100G+ gear can be used, but three more items must be checked over general switches: firmware version and compatibility, per-port testing (optical ports insert a module and test link), and licence／support status (used licences may not transfer). The method is in the used procurement guide.

Next step

Note: this article is a technical primer and selection framework; it does not guarantee any model, performance figure or compatibility. Actual performance depends on topology, firmware, tuning and workload; procurement conditions, warranty and liability follow the quote and the terms of service. Compatible optics are compatible products, not OEM modules; warranty generally one year.

FAQ

Q: Is InfiniBand required? A: Not necessarily. Within a few dozen machines, with an Ethernet-fluent team, a well-tuned RoCEv2 can work; the point is lossless and predictable, not which protocol.

Q: Is 800G the default for a new cluster? A: No. First check whether 400G is approaching its ceiling and whether the support is in place; 800G is a higher-density option, standardised under IEEE Std 802.3df-2024.

Q: Are compatible optics reliable? A: Usable, but choose the right model and write the conditions into procurement: platform, firmware version, replacement arrangement.