Design & build AI infrastructure

GPU systems

The three system classes an AI cluster is built from, the scale-up interconnect inside each, and how to choose. Ordinary servers with one or two accelerators are on Compute. Sources are vendor reference architectures and platform documents (NVIDIA Enterprise RAs and DGX user guides, AMD Instinct platform documents) and consortium specifications (UALink). Every figure here is generation-specific: GPU generations turn over roughly yearly, so recheck before you design. Exact SKUs, memory configurations, and prices vary by OEM; Cloudbit does not invent SKUs or publish unofficial pricing. Sources checked 5–6 Oct 2026.

System classes

Product names that recur in buying conversations: HGX is NVIDIA’s 8-GPU baseboard platform that OEMs build servers around; DGX is NVIDIA’s own turnkey system on it; MGX is NVIDIA’s modular server reference design, and the GB300 NVL72 rack is an MGX rack (Lenovo GB300 NVL72 datasheet (Mar 2026)). AMD’s equivalent building block is the OAM module on a UBB baseboard. The prior NVIDIA generation (H100, H200, B200) still has a maintained RA and is still widely deployed (NVIDIA HGX H100/H200/B200 RA, Aug 2026).

How to choose

• Model size against memory. If each model fits on one GPU, a PCIe-card or 8-GPU server serves it without a compute fabric. Models that need more memory than one GPU push you to an 8-GPU scale-up domain; training at scale pushes you to a compute fabric, and the largest models to a rack-scale domain.
• Training vs inference mix. NVIDIA’s HGX RA allows dropping the compute fabric for pure inference, at the cost of training and hybrid use later (Network separation).
• What the hall can deliver. An air-cooled 8-GPU node is about 14.5 kW per 10 RU; a liquid-cooled NVL72 rack is provisioned with up to eight 33 kW power shelves. A hall without liquid cooling rules out the rack-scale class (Facility readiness).
• East-west bandwidth per GPU. Current NVIDIA RAs range from about 200 Gb/s per GPU (RTX PRO) to 800 Gb/s per GPU (HGX B300); this sets NIC, optics, and switch counts (Fabric sizing).
• Software lock-in. The accelerator vendor sets the driver, runtime, and collective library. Moving between vendors means revalidating the stack (Software stack).

Scale-up interconnects

Scale-up links GPUs inside a server or rack with memory-semantic bandwidth well above the scale-out NIC. Traffic inside the scale-up domain never touches the AI cluster network.

Open scale-up standards are early and fast-changing. Treat them as a roadmap question for the next refresh, not a buying criterion for current systems.

Reference designs and validated systems

NVIDIA publishes three Enterprise RA families and sizes each in scalable units (SUs): 4 nodes for RTX PRO and HGX, one rack for NVL72. RTX PRO scales to 32 nodes (256 GPUs), HGX to 128 nodes (1,024 GPUs), and NVL72 to 8 racks (NVIDIA design tenets, Sep 2026). DGX SuperPOD is NVIDIA’s larger turnkey pattern. AMD publishes platform-level documents for its 8-GPU systems and builds Helios on OCP rack specs. NVIDIA-Certified Systems is NVIDIA’s catalog of qualified partner servers (NVIDIA-Certified Systems). Certification reduces GPU, NIC, firmware, and driver integration risk; it does not replace facility power and cooling design.

GPU systems design / build / manage

Design

• Write down the workload mix (train, fine-tune, serve) and the largest model per job before choosing a class
• Pick the class the facility can support today, or fund the facility work first (Facility readiness)
• Size growth in scalable units, not individual servers

Build

• Buy from a validated or certified system list, and record the exact GPU, NIC, BMC, and firmware baseline per node in DCIM (Ops: DCIM)
• Burn in each node and each scale-up domain before it joins the cluster

Manage

• Track GPU health and error telemetry per node, and drain nodes on repeated errors
• Plan the next generation’s facility envelope at least one refresh ahead

Software stack

The layers between bare GPUs and a production AI service, by vendor and open component. Identity and privileged access are on Security; the monitoring stack is on Ops. Sources: NVIDIA Enterprise RA software pages, NVIDIA and AMD GPU Operator documentation, NCCL and RCCL documentation, Kubernetes documentation, and the AMD MI355X brochure. Product names and bundles change often. Sources checked 5–6 Oct 2026.

Layers

Support entitlements are per vendor and per layer. NVIDIA AI Enterprise needs a license for each GPU, sold as a subscription, on consumption through cloud marketplaces, or as a perpetual license with five years of required support (NVIDIA AI Enterprise licensing guide); no on-prem price is cited here. Budget and track entitlements alongside hardware support.

Control-plane sizing

Plan management and control-plane nodes up front; they are easy to forget in a GPU BOM. NVIDIA’s RTX PRO RA gives a worked example of seven nodes: two for Base Command Manager (HA), two Slurm heads, and three for the Kubernetes control plane (NVIDIA RTX PRO RA). Your count depends on which schedulers you run.

Software stack design / build / manage

Design

• Choose Slurm, Kubernetes, or both from the workload mix, and size the control plane to match
• Pin a supported driver, runtime, collective-library, and operator combination from the vendor matrices before the pilot

Build

• Deploy drivers and runtimes through an operator or image pipeline, not by hand per node
• Record software baselines next to firmware baselines in the same runbook

Manage

• Upgrade driver, firmware, and fabric software in planned waves, and rerun collective benchmarks after each wave
• Track entitlement renewals per GPU and per layer
• Keep model-serving SLOs in the same runbook as firmware and fabric upgrades

Facility readiness

AI clusters fail when GPUs land before the hall can feed and cool them. Each check below states what an AI cluster asks of the facility, with dated vendor figures. The system design is on the owning page: power chain and distribution on Power, liquid cooling and the cooling plant on Cooling, cabinets and load ratings on Racks, and structural floor loading on Facility. All figures are for current generations (DGX B300, GB300 NVL72, Instinct MI355X) and will change with the next. Sources checked 5 Oct 2026.

Density envelope

Start from the system class, then nodes or racks per scalable unit, then the number of units you will grow to. An 8-GPU air-cooled node such as the DGX B300 draws 14.5 kW in 10 RU, so rack power is nodes per rack × node power plus switches (DGX B300 user guide, Oct 2026). A rack-scale NVL72 is one unit, provisioned with up to eight 33 kW power shelves (Lenovo GB300 NVL72 datasheet (Mar 2026)); get the rack’s design power from the OEM. Compare these figures with the hall’s design kW per rack; the gap usually decides between retrofitting, a dedicated AI hall, or colocation.

Power per rack

Ask the power owner for feed count and redundancy per rack, whether busway or high-amperage whips are needed, and whether branch circuits are metered per rack. Distribution design is on Power.

Liquid cooling

Rack-scale systems are liquid-cooled: NVIDIA describes the GB300 NVL72 as fully liquid-cooled (NVIDIA GB300 NVL72 product page). Lenovo’s implementation cools CPUs, GPUs, and switches with warm water (18–45 °C, PG25 or DI water), uses supplemental airflow for the remaining heat, blind-mate manifolds, and internal leak detection (Lenovo GB300 NVL72 datasheet (Mar 2026)). 8-GPU systems come in both forms. The DGX B300 is air-cooled at 10–30 °C inlet and about 1,500 CFM per node (DGX B300 user guide). AMD’s MI355X platform specifies direct liquid cooling with PG-25 coolant, a maximum liquid inlet of 43 °C, and a recommended 2.1 l/min per OAM module, with cold plates from qualified vendors (AMD MI355X brochure). Give the cooling owner the coolant type, inlet temperature, flow per device, leak-detection integration, and the residual heat that still goes to air. CDU and secondary-loop design are on Cooling; liquid-ready provisions for a hall built on air are on Cooling: Liquid-ready design.

Floor loading and rack size

A DGX B300 weighs 168 kg with PSUs, or 123 kg in the busbar model (DGX B300 user guide); the DGX B200 is up to 142.4 kg (DGX B200 user guide). Add the cabinet, PDUs or busbar, and cabling for a loaded-rack weight, and check both the cabinet rating (Racks: Load ratings) and the floor (Facility: Floor loading). AMD Helios uses the OCP Open Rack Wide spec, a different form factor from a standard 19-inch cabinet (AMD Helios PR); check row layout and aisle widths (Racks: Height, width and depth).

Facility readiness design / build / manage

Design

• Get written kW per rack, cooling capacity, and floor-load figures from the facility or colo provider before choosing a system class
• Plan utility upgrades, generator capacity, and facility water capacity for the final scalable-unit count, not the pilot

Build

• Commission liquid loops, CDUs, and leak detection, and test leak alarms end to end, before GPU racks run jobs
• Verify branch power monitoring per rack and alarm thresholds against the vendor power figures

Manage

• Trend rack power and coolant temperatures against the design envelope as utilization rises
• Revisit PUE, WUE, and cost-per-token metrics as density rises

AI cluster networking

The networks inside an AI cluster: the scale-out compute back-end that carries GPU-to-GPU RDMA traffic between servers, the converged front-end and storage networks, and their separation. The back-end is a routed Clos like the one in Network: Fabric design, so underlay, ASN, oversubscription, and ECMP basics are not repeated here. Lossless Ethernet mechanics (PFC, ETS, ECN, DCQCN, PFC watchdog) are defined on Network: Lossless Ethernet; this section covers only AI-specific tuning. MTU and encapsulation overhead are on Network: MTU; optics on Network: Optics, DAC and AOC, and Optics roadmap; OOB design on Network: OOB; and DC switching leaders on Network: Market leaders. Scale-up (NVLink and equivalents) is in Scale-up interconnects. Sources are standards bodies (UEC, IBTA), OEM reference designs (NVIDIA, Cisco, Juniper/HPE, Arista, AMD), and published operator and academic papers. Reference architectures change with every GPU and NIC generation, and the Ultra Ethernet spec is under active revision, so recheck before you design. Sources checked 5 Oct 2026.

Network separation: back-end, storage, front-end, OOB

Every reference design checked splits a training cluster into several physical networks. NVIDIA’s Enterprise RAs separate east-west GPU compute traffic from north-south traffic, which carries customer uplinks, storage connectivity, and support and management services (NVIDIA design tenets). Cisco and Juniper describe the same split, with a separate OOB network in each (Cisco AI clusters WP, Juniper AI cluster design).

Why separate. Meta’s reasons for a dedicated back-end were that the two networks would “evolve independently”, that the two traffic types have “very different and often conflicting requirements”, and that physical separation gave “the ideal network environment for latency-sensitive RoCE operations” (Meta, SIGCOMM 2024 §3.4). The UE spec makes the same trade: two networks cost more, but give “a clean separation of traffic (i.e., no interference) and design” (UE spec v1.0.2 §1.5.1.2). In practice the back-end can run lossless or sprayed RDMA with collective-tuned buffers, while the front-end keeps filtering, policy, and encapsulation. A shared back-end can still need tenant isolation: Cisco documents a rail-optimized VXLAN EVPN back-end for GPU-as-a-service (Cisco GPUaaS EVPN, Mar 2026); lossless RDMA over VXLAN has extra requirements, covered on Network: Lossless Ethernet.

No compute fabric at all. NVIDIA’s RTX PRO RA says the compute network is not necessarily required for inference (NVIDIA RTX PRO RA topologies (May 2026)), and its HGX RA allows pure-inference deployments where each model fits on one GPU to omit the compute fabric. The cost is that training and hybrid workflows are then not possible, and adding the fabric later means significant downtime and reconfiguration (NVIDIA HGX RA). If you pilot this way, reserve the NIC slots, switch ports, rack space, and cable pathways for a compute fabric from day one (Pilot).

Compute back-end fabric

The back-end starts where the scale-up domain ends: 8 GPUs in an HGX or UBB server, 72 in an NVL72 rack. Collective libraries (NCCL, RCCL) use the scale-up links first and the back-end NICs for everything that crosses a server or rack. In current NVIDIA RAs the per-GPU back-end bandwidth runs from up to about 200 Gb/s (RTX PRO: 4 × 400G SuperNICs per 8 GPUs; about 100 Gb/s with the 200G NIC option) to 800 Gb/s (HGX B300: one ConnectX-8 per GPU) (NVIDIA RTX PRO RA, NVIDIA HGX RA). That figure, times GPUs per scalable unit, sets leaf count, optics count, and cable count. PCIe-card designs without NVLink behave differently: GPU-to-GPU traffic inside the server crosses PCIe, two GPUs share each back-end NIC, and rails are per NIC, not per GPU. NVIDIA’s RTX PRO RA still describes its compute fabric as rail-optimized through the SuperNICs (NVIDIA RTX PRO RA topologies (May 2026), Lenovo RTX PRO 6000 product guide LP2263 (Jul 2026)). The back-end design decisions are, in order: whether you need it (above), topology and plane count, sizing, transport, and the congestion and load-balancing mode. Optics reach and form factor are on Network: Optics; most GPU-to-leaf links use breakout optics or AOC/DAC (DAC and AOC), and 200G-per-lane generations are tracked on Optics roadmap.

Rail-optimized and multi-plane topologies

Rail-optimized. NIC n of every server in a block connects to leaf n, so each leaf (a “rail”) holds the same GPU position across all servers. Juniper: “rail Nth connects all GPU in position Nth on all the servers, to leaf node Nth”. A “stripe” (NVIDIA: scalable unit) is the replicated block of rails and servers. Collective libraries move data over NVLink to the GPU on the destination’s rail first (NVIDIA PXN), so most traffic stays on one leaf. That alignment depends on an NVLink path inside the server; in PCIe-only servers, cross-rail traffic inside the server crosses PCIe instead. Cisco: rail-optimized design allows “single-hop forwarding through the leaf switches, without the traffic going to the spine switches” (Juniper AI cluster design, Cisco AI clusters WP (Jun 2025)). Cross-rail and cross-stripe traffic still uses the spine, and the spine is sized for it.

Sizing: non-blocking and where to oversubscribe

Why non-blocking (1:1). Training traffic is a small number of synchronized, line-rate elephant flows. A collective finishes only when its slowest flow does, so one congested uplink stalls every GPU in the job. NVIDIA builds the compute fabric as “a full non-blocking fat tree”. Cisco’s leaves have “an equal number of downlinks to GPU nodes and uplinks to the spine switches”. Juniper’s designs “follow customer use cases and GPU vendor recommendations of 1:1 oversubscription”, and in its JVD testing a 2:1 fabric “no longer has enough capacity” for all-inter-stripe traffic (NVIDIA HGX RA, Cisco, Juniper design, Juniper JVD). The leaf-ratio arithmetic is in Network: Oversubscription.

Where oversubscription is acceptable. 1:1 is a per-domain target, not a law. Meta builds each two-stage “AI Zone” non-blocking, oversubscribes the layer between zones “by design”, and uses topology-aware scheduling to keep cross-zone traffic low (Meta §3.3). If you oversubscribe anywhere, do it above the scalable unit, and make the scheduler aware of the boundary.

Sizing inputs. Size the back-end per scalable unit: count the back-end NIC ports per SU (GPUs × links per GPU in 1:1 designs; NICs × ports per NIC where GPUs share a NIC), and 1:1 sets an equal uplink capacity. NVIDIA’s tested maximums are 8 SUs, 32 nodes, and up to 256 GPUs for RTX PRO; 32 SUs, 128 nodes, and up to 1,024 GPUs for HGX; and 8 racks for NVL72 (NVIDIA design tenets (Sep 2026)). Each plane in the HGX dual-plane design scales to 1,024 × 400G interfaces (NVIDIA HGX RA). Switch radix decides whether the target fits in two tiers or needs a third.

RoCEv2 vs InfiniBand vs Ultra Ethernet

Congestion control and load balancing at AI scale

PFC, ECN, and DCQCN in practice. The mechanisms, the ECN-below-PFC threshold rule, and the PFC watchdog are defined on Network: Lossless Ethernet; this is AI-specific experience. Juniper’s AI congestion-control guide gives the thresholds it tested as a starting point and tunes them from PFC and ECN counters (Juniper); Cisco’s AI/ML blueprint does the same for Nexus (Cisco blueprint, Mar 2024). Treat any vendor’s numbers as lab-specific. Meta is a counterexample. After DCQCN tuning problems and firmware regressions it ran 400G with PFC only, and handled congestion in the collective library with receiver-driven clear-to-send admission (Meta §5.2).

Adaptive routing and packet spraying. Static ECMP hashing fails on AI traffic because a few long, line-rate flows collide on the same uplink. Meta: “ECMP rendered poor performance” on its low-entropy training traffic (Meta §4.1). The options trade reordering against balance. Flowlet DLB moves a flow only across an idle gap long enough to keep packets in order (Juniper). Per-packet spraying balances best but reorders packets, so the receiving NIC has to place out-of-order data. NVIDIA Spectrum-X does this with Spectrum switches choosing “the least-congested port … on a per-packet basis” and the SuperNIC restoring order (NVIDIA, May 2023, Oct 2024). Cisco validated ECMP, DLB per-packet, and DLB flowlet modes on Nexus 9000 (Cisco AI clusters WP). UET builds spraying into the transport. The sender varies the entropy field per packet over plain ECMP, so the reordering logic sits at the endpoint and does not depend on a matched switch and NIC from one vendor (UE spec v1.0.2 §3.6.5.2). Switch-side adaptive spraying from any vendor is outside the UE spec.

Ultra Ethernet status (fast-changing)

UEC published Specification 1.0 on 11 Jun 2025. Maintenance releases followed: 1.0.1 (5 Sep 2025), 1.0.2 (28 Jan 2026, corrections to the CMS congestion-control algorithms), and 1.0.3 (16 Jul 2026), the current version (UEC spec history). 1.0.3 adds 200 Gb/s per-lane signalling and a negotiation type for boolean features, plus corrections; UEC states that its clarifications do not affect behaviour, compliance, or interoperability, while corrections are required for conforming implementations (UE 1.0.3 release notes). Link-layer retry (LLR), credit-based flow control (CBFC), and packet trimming are all optional. UEC’s chair lists flexible congestion management, small-message performance, scale-up transport, and in-network collectives for the next iteration (UEC chair letter).

Compliance. UEC publishes a self-attestation compliance kit: profile matrices that mark each feature mandatory, conditional, or optional, plus Transport and PHY/link-layer checklists and test-bed recommendations (UEC compliance). There is no third-party certification or logo program in the sources checked. Treat “UEC-compliant” or “UEC-ready” as a vendor claim until the vendor gives you its completed checklist, the profile it implements, and the optional features it supports. UEC’s Member Declarations page is a register of patent declarations, not compliance claims.

Vendor landscape (categories)

These are categories, not rankings. All Ethernet options below can run RoCEv2 today. Feature names are vendor terms, and comparable features are not necessarily interoperable. Overall DC switching positions are on Network: Market leaders.

AI NICs. The NIC decides much of the transport behaviour, so choose it with the switch. NVIDIA ConnectX-8 and BlueField-3 SuperNICs are the back-end NICs in NVIDIA’s RAs (NVIDIA HGX RA). AMD Pensando Pollara 400 is a PCIe Gen5 x16 NIC with up to one 400G port, marketed as “UEC ready”, with packet spraying, in-order delivery to the GPU, selective retransmission, path-aware congestion avoidance, and a programmable P4 pipeline; AMD positions it for back-end or front-end use (AMD Pollara 400 brief). Ask every NIC vendor for its UEC self-attestation checklist before relying on a “UEC ready” claim.

Product generations in these tables change often (800G to 1.6T ports, 200G/lane SerDes, co-packaged optics, PCIe Gen6 NICs). Recheck model capabilities and UEC claims at design time. No pricing is shown because none is publicly published.

AI cluster networking design / build / manage

Design

• Decide whether you need a compute fabric at all. If you pilot without one, reserve NIC slots, leaf ports, rack space, and pathways to add it later.
• Separate compute back-end, front-end/storage, and OOB physically. Collapse them with VLANs only at small scale, and only with a documented plan to split later.
• Build the back-end rail-optimized and 1:1 inside each stripe or scalable unit. Decide deliberately where, if anywhere, you oversubscribe (between zones), and pair that with topology-aware scheduling.
• Choose single- vs multi-plane from the SPOF tolerance, radix (tier count), and the breakout optics the NIC supports (Network: Optics)
• Pick the transport per cluster (RoCEv2 lossless, InfiniBand, or UET) and write down the loss model, congestion control, and load-balancing mode end to end. Mixing approaches on one fabric is not a design.
• Size the back-end L3 domain, NIC addressing, MTU (Network: MTU), and the DSCP-to-traffic-class plan before cabling. They have to match on every NIC and switch.
• Treat optics and cabling as first-class design, not an afterthought to the GPU order: count transceivers and trunks per scalable unit (Racks: MPO trunks)

Build

• Validate rail cabling (NIC n → leaf n) from LLDP against the source of truth before handover. One swapped rail quietly adds spine hops.
• Apply the QoS/PFC/ECN or UET traffic-class config from a template, and confirm under induced incast that ECN marks before PFC triggers (Lossless Ethernet)
• Run collective benchmarks (all-reduce, all-to-all) at full stripe and cross-stripe scale. Compare against the design bus bandwidth, and retest with one leaf, one spine, and one plane failed.
• Check that link-flap and error behaviour is acceptable (FEC counters, optics DOM) on every GPU-facing port before jobs land

Manage

• Track PFC pause duration, ECN/CNP (or trim) rates, drops, and per-uplink load imbalance per leaf and per plane
• Track job-level collective completion time and correlate slow jobs with specific links. Isolate flapping or erroring links quickly, because one bad link slows the whole job.
• Recheck congestion-control and load-balancing behaviour after every NIC firmware and NOS upgrade (Meta saw a DCQCN regression from a firmware change)
• Review UEC spec revisions, the compliance kit, and vendor feature support at least every six months

Phased build

Grow in scalable units: a replicated block of servers, rails, and storage connectivity (4 nodes in NVIDIA’s HGX and RTX PRO RAs, one rack for NVL72; NVIDIA design tenets). Each phase ends with a gate. DCIM and monitoring are on Ops.

Pilot

• Define the workload mix (train vs fine-tune vs serve) and data residency constraints
• Pick a certified or validated small-node BOM; validate the power and cooling envelope in a cage or lab row
• Stand up the driver, runtime, and serving path; measure utilization and thermals
• Plan control-plane nodes up front (Control-plane sizing)
• Document serials, firmware baselines, and fabric topology in DCIM
• Gate: inference-only without a compute fabric is a valid pilot only if the NIC slots, switch ports, and cabling for a later compute fabric are reserved (Network separation)

Production

• Freeze a repeatable rack elevation and cabling standard
• Commission liquid CDUs, leak detection, and power branch monitoring
• Harden OOB and Redfish; integrate monitoring for GPU, NIC, and CDU health
• Establish model registry, secrets, and backup/DR for critical datasets
• Gate: collective benchmarks pass at full scalable-unit and cross-unit scale, including with one leaf or plane failed (Fabric build)

Scale

• Grow by scalable units; fabric and storage scale with GPU count
• Plan utility upgrades, generator fuel, and facility water capacity early (Facility readiness)
• Automate provisioning and firmware waves; keep certified component sets in lockstep
• Revisit PUE, WUE, and cost-per-token metrics as density rises