Networking

Fabric design

Routed Clos underlay sizing and protocol choices. Overlay design is covered in EVPN/VXLAN. AI back-end (GPU-to-GPU) fabrics and GPU systems are on AI Factory (AI cluster networking). Each domain’s side of the fabric is on its own page: host NICs and bonding on Compute, cluster traffic on HCI, SAN transports and paths on Storage, and backup isolation and replication on Backup. Port counts in the examples are hypothetical round numbers, not specific products. Sources checked 5 Oct 2026.

3-stage vs 5-stage Clos

Terminology: RFC 7938 uses Tier 1 / Tier 2 / Tier 3 and calls a Tier 2 + Tier 3 block a “cluster”. Vendor docs say super-spine / spine / leaf and POD. “5-stage” counts the devices a host-to-host flow crosses (leaf, spine, super-spine, spine, leaf), not the number of tiers (RFC 7938 §3.2, Juniper 5-stage guide).

Notation: a leaf has D downlinks and U uplinks, one link to each spine. A spine has P_s ports. In 5-stage, each spine splits ports into d_s down and u_s up, and a super-spine has P_ss ports. Spine #k of every POD connects to super-spine plane k.

With equal-speed links, ECMP width between two leaves equals the number of top-tier devices (RFC 7938 §6.1). In 5-stage, end-to-end oversubscription is the product of the leaf ratio and the spine ratio.

eBGP underlay (RFC 7938)

RFC 7938 (Informational, Aug 2016) is an operator design document, not a protocol spec. Its model: eBGP on every fabric link, no IGP, and AS_PATH loop detection that keeps path hunting in check.

4-byte private ASNs. RFC 6996 reserves 4200000000–4294967294 for private use (vs 1,023 in 64512–65534), and four-octet ASN support is RFC 6793. That removes the reason RFC 7938 gave for reusing ASNs. Strip private ASNs at the border toward the WAN (RFC 7938 §5.2.4).

BGP unnumbered. This is an implementation pattern, not an RFC. Peers form over IPv6 link-local addresses learned from router advertisements, and IPv4 NLRI is carried with an IPv6 next hop per RFC 8950 (Nov 2020, obsoletes RFC 5549). It removes per-link /31 planning (Cumulus, FRR). Use it where your NOS supports it, and confirm interop before mixing NOSes in one fabric.

ECMP. RFC 7938 §6.2 requires load-sharing across equal-length AS_PATHs from different ASNs, known as “multipath relax” or “multipath multiple-AS”. The knobs are bgp bestpath as-path multipath-relax (FRR), multipath multiple-as (Junos), and maximum-paths (NX-OS). Set the maximum path count to at least the number of spines.

BFD. RFC 7938 §7.1 notes that on point-to-point links, link-down with fast fallover catches most failures in milliseconds. BFD (RFC 5880, single-hop RFC 5881) adds detection of failures that leave the link up. OEM examples: NX-OS defaults to 50 ms × 3 and can offload BFD to line modules (Cisco). Juniper’s fabric guide uses 100 ms × 3 (Juniper). Use aggressive timers only where BFD runs in hardware.

eBGP vs IGP underlay

Both approaches are validated by major vendors. eBGP-only gives the strongest failure isolation and scales furthest. An IGP underlay is a reasonable choice for single-POD fabrics run by teams that already operate IS-IS or OSPF. No source gives a fabric size at which IGP flooding becomes the limiting factor.

Routed to the leaf, no overlay

RFC 7938’s own design has no overlay. Routing runs down to the leaf (Tier 3), so each server subnet ends at its leaf or leaf pair. The RFC gives the main benefit of L3-only designs as “confining L2 broadcast domains” (RFC 7938 §4.3). The underlay choices above apply unchanged, and MLAG/vPC still dual-homes hosts to the leaf pair. EVPN multihoming and a firewall that joins the overlay as a VTEP (Service insertion) need EVPN, so they are not available here.

L2 adjacency stops at the leaf pair. Anything that needs a shared subnet must stay within one leaf pair: VM live migration that keeps its IP address, cluster heartbeats that need L2, and appliance HA pairs with a floating IP and MAC. RFC 7938 §4.2 names “seamless mobility for virtual machines that require the preservation of IP addresses when a virtual machine moves to a different Tier 3 switch” as one reason operators kept L2. It notes that newer solutions meet such requirements “by using various overlay or tunneling techniques” (RFC 7938 §4.2).

Zone VRFs become VRF-lite on every link. Without EVPN, each zone VRF from Security’s policy model is carried hop by hop. Every leaf–spine link needs one sub-interface (or VLAN) and one routing session per VRF, so configuration grows with VRFs × fabric links. This is the same back-to-back VRF mechanism as RFC 4364 §10 option (a), where routers are attached by “multiple sub-interfaces, at least one for each of the VPNs” and the RFC says it “does not scale as well” as the alternatives (RFC 4364 §10). The way out is fewer zones in the fabric, with finer enforcement on hosts or hypervisors (Security: Microsegmentation).

Fits when the fabric is small, few workloads need L2 across racks, and the zone count is low. Otherwise use EVPN/VXLAN. A VLAN-based L2 spine-leaf (MLAG at both tiers, with STP as the backstop) is a different design and is not covered here.

Oversubscription

Leaf ratio = Σ downlink bandwidth ÷ Σ uplink bandwidth (RFC 7938 §3.2.2 states it as N/M ports at equal speed). Example: 48 × 25G = 1,200G down and 8 × 100G = 800G up, so 1,200 ÷ 800 = 1.5:1. Those 8 uplinks also set the spine count at 8. The ratio describes worst-case contention, not typical load. Microbursts from TSO/LSO senders and incast can congest a port even at low average utilization. The options are minimal oversubscription, reduced fan-in, or deeper buffers (Arista).

Lossless Ethernet: PFC, ETS, ECN, DCQCN

RoCEv2 is normally run on a no-drop traffic class, because RoCE NICs have traditionally recovered from loss with go-back-N retransmission, which is expensive (Mittal et al., SIGCOMM 2018). The same mechanics apply to storage (NVMe/RoCE), HCI (RDMA for HCI), and AI back-end fabrics (AI Factory: RoCEv2 vs InfiniBand vs Ultra Ethernet). Those pages decide whether to use RDMA and which platform priorities apply. This section defines the mechanisms. TCP transports (iSCSI, NVMe/TCP, NFS, SMB without RDMA) need none of it.

Failure modes. PFC trades drops for pauses that can spread. The DCQCN authors cite “head-of-line blocking and unfairness”. Microsoft’s production paper reports PFC-induced deadlock (“yes, it happened!”), RDMA transport livelock, and NIC pause-frame storms (Guo et al., SIGCOMM 2016). The no-drop class must be configured identically on every hop between the endpoints. One hop with the wrong mapping, or no headroom, drops RDMA traffic or pauses far more than it should. This is why storage and HCI keep RoCE on dedicated switches or on a small, contained set of leaves.

Over VXLAN. Underlay switches see only the outer header. Lossless RDMA across an EVPN/VXLAN overlay needs the VTEP to copy the inner DSCP into the outer header at encapsulation (the uniform model in RFC 2983), DSCP-based PFC on every underlay hop, and ECN carried between inner and outer headers per RFC 6040. Vendors document this, for example Cisco’s rail-optimized VXLAN EVPN back-end (Cisco GPUaaS EVPN), but support depends on the ASIC and NOS release. Lossy alternatives: Ultra Ethernet Transport is designed for best-effort fabrics, and some RDMA NICs offer RoCE modes that do not need switch PFC. Use them only where the storage or HCI platform supports them. The AI transport comparison is on AI Factory.

Fabric design / build / manage

Design

• Size for the end-state port count and choose 3-stage vs 5-stage up front. Spine radix sets the leaf ceiling, and super-spine radix sets the POD ceiling.
• Set oversubscription per leaf role (compute, storage, border) and compute the end-to-end ratio across tiers. Storage targets come from Storage.
• Pick the underlay deliberately: eBGP-only (RFC 7938) or IGP + iBGP EVPN. Document why.
• ASN plan: shared spine ASN, unique 4-byte private ASN per leaf (or leaf pair). Avoid allowas-in / as-override unless a repeat is unavoidable.
• Keep host subnets unsummarized inside the Clos, and summarize only at the border

Build

• Generate ASNs, loopbacks, and link addressing (or unnumbered interfaces) from a source of truth, not by hand
• Confirm ECMP width equals the spine count on every leaf (multipath-relax / multiple-as / maximum-paths)
• Enable BFD with timers the hardware can sustain. Test link-down and link-up-but-dead failures separately.
• Fail a spine, and in 5-stage a super-spine plane, under load before handover

Manage

• Alert on ECMP path count dropping below the designed spine count
• Track uplink utilization and drops/buffer events per leaf against the design ratio
• Track spine and super-spine port headroom so you know when to add a POD
• Re-check BFD flaps and BGP session churn after every NOS upgrade

EVPN/VXLAN

The overlay that runs on the routed Clos described in Fabric design. Standards: RFC 7432 (EVPN), RFC 8365 (EVPN for NVO/VXLAN), and RFC 7348 (VXLAN, Informational). Vendor behaviour is cited from Cisco NX-OS, Arista EOS, Juniper Junos, NVIDIA Cumulus Linux, and SONiC/FRR documentation. Platform limits change with each release, so recheck them against your target versions. DCI transport (dark fibre, DWDM, MACsec) is out of scope here. Sources checked 5 Oct 2026.

Route types

Symmetric vs asymmetric IRB (RFC 9135)

Distributed anycast gateway. RFC 9135 allows two forms. In the first, every leaf uses the same gateway IP and MAC. In the second, leaves share the IP but keep their own MACs, which are aliased with the Default Gateway extended community. If hosts depend on any form of MAC security, RFC 9135 says to use the shared anycast MAC. Cisco’s distributed anycast gateway uses the same MAC on every VTEP for every L2 VNI SVI, giving active/active first-hop routing “without a need to implement … HSRP or VRRP” (RFC 9135, Cisco design guide). Juniper documents both the anycast IP/MAC form and a virtual-gateway form (Juniper ERB).

EVPN multihoming vs MLAG

Mechanics. In all-active mode, every leaf on an Ethernet segment (ESI) forwards known unicast. When a link fails, withdrawing the Type 1 per-ES route moves every MAC on that segment in one update (mass withdrawal), instead of one withdrawal per MAC. Type 1 per-EVI routes let remote VTEPs load-balance to every leaf on the segment, including leaves that never learned the MAC (aliasing). The DF elected through Type 4 routes is the only leaf that forwards BUM onto the segment (RFC 7432 §8).

DF election. The RFC 7432 default, (V mod N), spreads load unevenly when VLAN IDs cluster and remaps VLANs whenever a PE joins or leaves. RFC 8584 (Apr 2019) added HRW and AC-DF, which excludes a PE whose attachment circuit is down. RFC 9785 (Jun 2025) added preference-based election with a “Don’t Preempt” (non-revertive) option. Defaults differ by vendor. Cumulus and Arista EOS use preference (default 32767, highest wins). Junos defaults to MOD and offers preference, with non-revertive from 24.2. SONiC defaults to modulo and also offers HRW and preference (Cumulus, Arista, Juniper, SONiC). Under RFC 8584, if any PE on the segment does not signal an algorithm, all of them fall back to modulo. Set the algorithm explicitly per segment, and test failover if a segment mixes vendors.

Split-horizon. VXLAN has no ESI label, so RFC 8365 §8.3.1 uses local bias. The ingress VTEP floods BUM to its own ports on the segment regardless of which leaf is DF. Egress VTEPs drop BUM whose tunnel source IP belongs to a peer on the same segment. RFC 9746 (Mar 2025) adds a split-horizon type field to the ESI Label extended community. That lets operators choose between ESI-label and local-bias filtering on encapsulations that support both (MPLSoUDP, MPLSoGRE, Geneve, SRv6). VXLAN still uses local bias.

One LACP partner toward the host. A host LAG split across two leaves comes up only if both leaves present the same LACP system ID, so the host sees a single partner. MLAG/vPC does this through the pair’s shared system MAC. With ESI-LAG it has to be configured: Juniper says “the same ESI and LACP system identifier should be configured on all links that make up a LAG bundle” and suggests deriving the system ID “from the last 6 bytes of that ESI”. In Cumulus, “The Ethernet segment MAC address is the LACP system identifier”, and the type-3 ESI is built from it (Juniper EVPN-LAG guide, Cumulus EVPN-MH). If the IDs don’t match, the host puts one member in a separate aggregator or suspends it. Host teaming modes are on Compute: Host bonding.

EVPN all-active wins on cabling, N-way redundancy, and multi-vendor standards. MLAG wins on overlay state, failover churn, and STP behaviour. Arista notes the two models interoperate in one fabric and frames the choice around brownfield vs greenfield, STP interaction, cabling, redundancy level, and single-homing needs. Cisco’s general design guide still presents vPC (including vPC fabric peering) as its main dual-homing method. Platform support for ESI multihoming changes with every release.

ARP/ND suppression and BUM handling

Suppression. A leaf that holds MAC-IP bindings from Type 2 routes answers ARP and ND locally (RFC 7432 §10). RFC 9161 notes that address-resolution flooding grows linearly with host count, and that proxying reduces it, “and in some cases, it can even be suppressed”. On a cache miss the request is still flooded, so silent hosts and moves still generate BUM. Implementations: NX-OS suppress-arp, which needs TCAM carving on some older platforms. Cumulus enables ARP suppression by default and requires it for EVPN multihoming. SONiC/FRR handle it in the kernel (neigh_suppress) (NX-OS, Cumulus, FRR).

RFC 8365 requires PEs with globally assigned VNIs to support ingress replication or PIM-SSM. PIM-SM and BIDIR-PIM are optional. No vendor source gives a VTEP count at which ingress replication becomes the bottleneck. Choose the mode up front from the expected BUM rate and VNI spread, because changing it later is disruptive.

Multi-site and DCI

RFC 8365 §10 defines two DCI groups. DCI using gateways terminates EVPN at each site’s gateways. With unknown-MAC and default routes, VTEPs hold only local state, and mass withdrawals stay inside the site. DCI using ASBRs keeps the DCI box simple, but VTEPs carry remote state, VNIs become downstream-assigned, and the ASBR’s next-hop rewrite “impacts the EVPN mass withdrawal route … adversely”. RFC 9014 (May 2021) specifies the gateway group in two forms:

In both models the gateway should advertise an Unknown MAC Route into the site, run proxy ARP/ND, and translate VNIs and tunnel source IPs at the boundary (RFC 9014 §3.5, §4.6). Vendor implementations include Cisco’s Multi-Site border gateways: up to four anycast BGWs per site, an eBGP site-external overlay, and ingress replication between sites (Cisco Multi-Site WP, 2022). Juniper offers Type 5, IPVPN, and VXLAN-stitching DCI (Juniper).

VNI/VRF design

VNIs. The VNI is 24 bits, which gives about 16M segments (RFC 7348). RFC 8365 §5.1.2 offers two service models: VLAN-based, with one VNI per EVI and its own RD/RT, and VLAN-aware bundle, with several VNIs per EVI identified by Ethernet Tag. Plan one L3 VNI per tenant VRF, globally unique VNIs inside a site, and translation at the DCI gateway (RFC 9014 §4.6). Which zones get a VRF, and which inter-VRF flows must cross a firewall, come from Security’s policy model. RFC 8365 RT auto-derivation puts a 2-octet ASN in the RT. In eBGP fabrics with a different ASN per leaf, set RTs explicitly or confirm how your NOS handles that field.

MTU and VXLAN overhead

Two numbers matter. The tenant MTU is what hosts, arrays, and HCI nodes set on their interfaces. The underlay MTU is what every fabric link must carry: the tenant MTU plus the encapsulation overhead below. Other pages set the tenant MTU for their traffic and link here for the overhead.

The figures assume no IPv4 options and no IPv6 extension headers. The inner FCS is not carried. Sizes are taken from the RFC 7348 §5 frame formats. “VTEPs MUST NOT fragment VXLAN packets”, and the receiving VTEP “MAY silently discard” fragments (RFC 7348 §4.3). Cisco sizes the underlay at 9216 to fit “maximum server MTU + 50 bytes VXLAN overhead” (Cisco design guide). Check whether your NOS’s MTU setting counts the Ethernet header (L2) or not (L3) before you compare numbers across vendors.

Encryption overhead on DCI. Encryption on the DCI adds its own overhead on top of any overlay. MACsec makes each frame bigger on the link without changing the IP MTU, so the carrier service has to accept the larger frame. IPsec tunnel mode reduces the IP MTU available inside the tunnel. Size the DCI path for tenant MTU + overlay + encryption, and clamp the TCP MSS where PMTUD can’t be relied on. The byte counts below are worked out from the field sizes in the cited specs.

IPsec padding varies with packet length, so plan for the upper figure. GRE, VXLAN, or SD-WAN headers carried inside the tunnel add to these numbers. Vendor SD-WAN overlays add their own headers, so check each vendor’s figure.

Overlay peering and the underlay

Underlay protocol, ASN plan, and 3- vs 5-stage scale are in Fabric design. This table adds only where EVPN peers in each model.

The EVPN BGP next hop is the remote VTEP. A spine that rewrites it to itself pulls VXLAN tunnels onto a device that is not a VTEP. RFC 8365 §10.2 describes the same next-hop rewrite breaking mass withdrawal at an ASBR. Inside a Cisco Multi-Site site, the overlay “always behaves like an iBGP deployment” even when the underlay is eBGP (Cisco Multi-Site WP).

Firewall and service insertion

This is the plumbing behind Security’s inter-zone policy. Security decides which zones exist, which inter-zone flows are inspected, and which are L3/L4-only (Security: policy model). Network builds one tenant VRF per zone and makes inspected flows reach the firewall. Firewall placement, HA mode, and sizing are on Security: Perimeter. In Cisco’s leaf roles, a service leaf is a leaf attached to “L4 – L7 devices” such as firewalls and load balancers, and a border leaf connects edge routers (Cisco design guide).

Per-VRF default route. The firewall can advertise the default to each zone VRF (eBGP per VRF), or the service leaf can hold a static default per VRF toward the firewall. Either way, it reaches the other leaves as an EVPN Type 5 route. Use one default source per VRF. A second default, for example an internet default from the border leaf leaked into the same VRF, gives a path around the firewall. If the default is static, track the firewall’s reachability so a dead next hop does not blackhole the VRF.

Symmetry for stateful devices. A stateful firewall drops a flow when it sees only one direction. Keep both directions on the same firewall, or the same state-sharing cluster. Common ways to break this: a route leak that carries one direction while the return follows the default through the firewall; the same zone attached to two firewall pairs (two service-leaf pairs or two sites) with ECMP or anycast between them; and active/active units without the vendor’s flow-ownership mechanism. In multi-site designs, pin each zone’s inspected traffic to one site’s firewall, or use a cluster that shares state across sites.

Attaching the firewall pair. Dual-home each firewall to the service leaf pair with EVPN multihoming (ESI-LAG) or MLAG/vPC (trade-offs above). Juniper’s guide uses an ESI connection toward the firewall. The HA pair’s floating IP and MAC move between leaves on failover, so the fabric sees an EVPN MAC move. RFC 7432 §15.1 declares a duplicate MAC after N = 5 moves within M = 180 s by default, then stops “sending and processing any BGP MAC/IP Advertisement routes for that MAC address until a corrective action is taken”. A flapping firewall or ADC pair can trip this, so test repeated failovers. Routing adjacencies over MLAG/vPC or ESI-LAG need explicit NOS support. Check your vendor’s guide, or use routed point-to-point links per VRF instead. Keep HA and state-sync links as the firewall vendor specifies.

Route-leak policy. In EVPN, routes leak between VRFs by importing another VRF’s route targets, usually filtered by a route map. Leak only for flows the policy model marks L3/L4-only, such as a shared-services VRF for DNS, NTP, and directory. Record each leak in Security’s inter-zone matrix (policy model), and enforce it with leaf ACLs or group-based policy. Any leak not in the matrix is a bypass of the firewall, and it is reviewed as drift (Security: drift).

EVPN/VXLAN design / build / manage

Design

• Use symmetric IRB with a distributed anycast gateway (shared IP and MAC), and one L3 VNI per tenant VRF. Size the leaf host-route FIB for the largest VRF.
• Write down the VNI, VRF, and RT scheme before the first leaf ships, starting from Security’s zone-to-VRF mapping (policy model). Set RTs explicitly if leaf ASNs differ.
• Choose EVPN all-active or MLAG per rack type, using the state vs failover trade-off above. For ESI, fix the DF algorithm per segment.
• Choose ingress replication or underlay multicast up front from the expected BUM rate and VNI spread
• Set the underlay IP MTU to at least the tenant MTU + 50 (IPv4 outer) or + 70 (IPv6 outer), with headroom. Never rely on fragmentation.
• For multi-site, use a gateway model (RFC 9014 decoupled or integrated) rather than one stretched EVPN domain when you need failure and BUM containment
• Build one VRF per Security zone, with a single default per VRF toward the firewall at the border or service leaf. Allow route leaks only where the inter-zone matrix marks a flow L3/L4-only (Service insertion).

Build

• Confirm spines preserve the EVPN next hop and, in eBGP-only fabrics, retain all route targets
• Ping between VTEP loopbacks at full tenant MTU plus overhead with DF set, over every spine
• Fail one leaf of each ESI pair and check mass withdrawal, aliasing, and DF takeover under load. Repeat for an MLAG pair.
• Check that ARP/ND suppression is active per VNI, and that a silent host still resolves
• Confirm Type 5 routes from the border leaf carry the correct VRF and Router’s MAC to every leaf
• Trace one flow per inspected zone pair in both directions, and confirm both pass the same firewall. Fail the firewall pair over several times in a row, and check that the floating MAC does not trip duplicate-MAC detection.

Manage

• Track EVPN route counts per type (2, 5, 1/4) and host-route FIB use against platform limits
• Alert on MAC mobility churn and duplicate-MAC/IP detection events
• Watch BUM rate per VNI and ingress-replication flood-list length as VTEPs are added
• Recheck ESI multihoming and DF behaviour after every NOS upgrade. Vendor support changes per release.
• Compare VRF route-target imports and redirect rules with the inter-zone matrix. Any leak not in the matrix is drift (Security).

Optics & interconnect

The transceivers, direct-attach and active cables, and application loss budgets that run on the passive plant. The plant itself (fiber type, trunks, MPO polarity, panels, certification, labeling, as-builts) is on Racks: Structured cabling plant. The hand-off between the two pages is a per-link record of fiber type, connector type and count, polarity method, length, and designed loss: Racks supplies it, and this section checks it against the PMD. Reaches and channel insertion loss below are IEEE minimums as summarized by the TIA Fiber Optics Technology Consortium (FOTC) or as stated on OEM module datasheets. The IEEE texts were not read in full. Products may be rated differently from the PMD they implement. For example, Cisco rates its 400G LR4 module to 10 km against the 6 km 400GBASE-LR4-6 PMD, and NVIDIA’s 100G-PAM4 multimode line has a 50 m maximum. Optics generations change every 12 to 18 months, so recheck this section about every 6 months. No pricing. Sources checked 5 Oct 2026.

PMDs by reach class

Twin-port 800G modules in 2 × FR4 form (two duplex connectors) are also sold by switch OEMs. They were seen only in reseller listings this pass, so no connector or reach is given here.

Loss budgets and the plant hand-off

Check designed loss against the channel budget. Racks builds each link’s designed loss from component allowances (Racks: Fiber media). Compare it with the channel insertion loss of the PMD you intend to run, including the next generation. The 100G-per-lane multimode PMDs leave little room. Racks’ worked example of a 50 m OM4 link with two mated pairs at standard maximum allowances designs to 1.65 dB. That passes 400GBASE-VR4 (1.7 dB) and 400GBASE-SR4 or 800GBASE-SR8 (1.8 dB) with 0.05 to 0.15 dB of margin, and a third mated pair at 0.75 dB fails all of them. Multimode links with a cassette at each end therefore need low-loss components designed to the vendor’s maximum figures, or a move to singlemode.

Reflectance on short-reach singlemode. The DR and FR PMDs have loss budgets closer to multimode than to classic long-reach singlemode, and their budgets assume a limit on discrete reflectances. Fluke Networks’ worked examples show that at −37 dB connector reflectance only one connector fits in a short-reach singlemode channel, while at −49 dB ten do, and its 400GBASE-DR4 examples put the allowed channel loss at 3.0 dB or 2.7 dB depending on the reflectance conditions (Fluke Networks, Oct 2025). So the hand-off record needs connector reflectance (APC for parallel singlemode, per Racks: MPO trunks) and the OTDR reflectance results from Racks: Testing and certification, not only total loss.

Breakouts set the patch field. A 400GBASE-DR4 port breaks out to four 100GBASE-DR ports on duplex LC. Cisco supports this to QSFP28 100G-DR, -FR and -LR modules for links up to 500 m. A 400GBASE-SR8 port breaks out to 8 × 50GE or 2 × 200GE (Cisco). Decide where the breakout happens (harness, MPO-to-LC cassette, or at the far-end panel) before the trunks are ordered. Each choice changes the mated-pair count and the trunk base on Racks.

Multimode vs singlemode. At 100G per lane, multimode reaches 50 to 100 m on OM4, which covers in-row and most in-hall links but not hall-to-hall or MDA-to-entrance runs. The adopted P802.3dj objectives for 200G-per-lane PHYs list only singlemode and copper twinax media, with no multimode objective (P802.3dj objectives, Mar 2024). Some colos also deliver only singlemode cross connects (Edge & carrier). A new backbone plant that has to survive the 200G-per-lane generation should therefore be singlemode. Whether multimode stays in the horizontal is a per-link-class decision made jointly with Racks.

DAC, active copper, and AOC

Direct-attach and active cables are transceivers with a fixed cable, not plant. They bypass the patch panels and do not appear in the cabling certification.

The objective for 200G-per-lane passive twinax is about 1 m. If that holds in the final standard, passive DAC at that generation becomes an in-cabinet medium at best. Switch placement that relies on DAC to reach adjacent cabinets (Racks: switch placement) will need active copper or optics.

Roadmap and power

• Lane generations. 50G PAM4 per lane (400GBASE-SR8, 802.3cm). 100G per lane (802.3db-2022, and 802.3df-2024, approved 16 Feb 2024 per IEEE 802.3df). 200G per lane is P802.3dj, which covers 200G, 400G, 800G, and 1.6T PHYs. It went to IEEE SA ballot after conditional approval on 3 Feb 2026, and approval is expected by December 2026 (IEEE SA P802.3dj).
• 200G-per-lane singlemode objectives. 800G over 4 SMF pairs at 500 m and 2 km, over 4 wavelengths at 500 m, 2 km, and 10 km, and over 1 wavelength at 10 km. 1.6T over 8 SMF pairs at 500 m and 2 km. A plant built for 400GBASE-DR4 (8 fibers) carries an 800G 4-pair PHY on the same fiber count (P802.3dj objectives).
• Module power is rack power. NVIDIA lists its 800G twin-port OSFP transceivers at 17 W maximum and its 400G QSFP112/OSFP transceivers at 9 W maximum (NVIDIA). As a hypothetical, a 32-port leaf fully populated with 17 W optics carries up to 544 W of optics on top of the switch. Give that figure to Power and Cooling per cabinet.
• Linear pluggable optics (LPO). The LPO MSA released its specification on 25 Mar 2025 for “100 Gb/s, 200 Gb/s, 400 Gb/s and 800 Gb/s Ethernet parallel single-mode links” at 100G per lane, aiming at lower power, cost, and latency (LPO MSA). Linear modules rely on the host SerDes for equalization, so interoperability depends on the host as well as the module. Qualify host and module together.
• Co-packaged optics (CPO) move the optical engine next to the switch ASIC. That removes the pluggable as a field-replaceable unit, which changes sparing and failure handling. Treat it as a fast-changing category and recheck at design time. No product specs are given here.

Optics design / build / manage

Design

• Choose the PMD per link class from the length, loss, and reflectance that Racks hands over, and check the next generation’s budget too
• Plan the roadmap (400G and 800G uplinks now, 200G per lane next) together with the trunk base and fiber type on Racks
• Decide where breakouts happen before the trunks are ordered
• Standardize on a small set of PMDs and form factors, and give per-cabinet optics power to Power and Cooling

Build

• Before turn-up, check each link’s designed loss and certified results (Racks: certification) against the PMD’s channel insertion loss
• Inspect and clean module and cord end-faces before inserting optics
• At turn-up, record DOM receive power and FEC corrected and uncorrected counters per port as the baseline
• Record the PMD and module part number per port in the source of truth

Manage

• Trend DOM (Tx/Rx power, bias, temperature) and pre-FEC error rates against the turn-up baseline
• Hold spares per PMD and form factor, not per speed
• Recheck the PMD table, 802.3dj status, and LPO/CPO maturity about every 6 months

Out-of-band management

This is the management network you use when the production network is down, misconfigured, or compromised. It carries switch and router management ports, device consoles, server BMCs, switched rack PDUs, rack transfer switches, and the cards of UPS units that feed only IT racks. Facility power devices stay in the facility OT zone; the full split is on Security: PDU and UPS cards. Sources: CISA’s Dec 2024 hardening guidance, Cisco’s Out of Band Best Practices white paper (2025), console-server vendor documentation, and colo provider service documentation. Cisco states its paper describes “not Cisco-validated designs”. Perimeter policy is on Security: Perimeter, and admin identity, PAM, and BMC hardening on Security: Identity. Sources checked 5 Oct 2026.

Design principles

Physically separate. CISA: “Use an out-of-band management network that is physically separate from the operational data flow network.” Device management should “only come from the out-of-band management network”, and that network should not allow “lateral management connections between devices” (CISA, Dec 2024). Cisco adds that management circuits must stay “logically and physically isolated from the devices being managed”: management traffic “should never traverse the same devices it is intended to monitor or control”. Cisco also discourages L2VPN or L3VPN transport for management (Cisco OOB best practices). It follows that a management VLAN or VRF carried on the production fabric is in-band management with segmentation, not OOB, because it fails when the fabric fails.

Separate addressing and routing. Cisco recommends a dedicated management VRF, a separate VRF-aware routing process instead of one shared with production, no default route on management interfaces “where feasible”, static IPs on every management and managed-device interface, simple routing, and no redistribution unless it is filtered. Use an OOB address block that does not overlap production, so the OOB stays usable when production routing is what broke.

No dependency on production. Cisco: TACACS+, syslog, observability, and automation should be “dedicated (Recommended) or reachable within Management network”. If MFA is unreachable, devices must “fallback to ‘local account’”. Add DNS and NTP to the list, or make break-glass procedures work with IP addresses and local clocks. Common traps: AAA servers running as VMs on the cluster you are trying to recover, jump hosts in a production VRF, and an OOB internet uplink that goes through the production firewall.

Power telemetry conduit. EPMS and DCIM collectors in the monitoring DMZ reach the OOB-hosted PDUs, rack ATS, and UPS cards through one filtered conduit. The conduit rules are on Security: PDU and UPS cards, and collection paths on Ops.

Access order. Cisco’s model: in-band first, then the dedicated management interface over the management network, then the console. Only the console still works after a bad config push that takes down the management interface, or when a device will not boot its NOS.

Cellular / LTE backup access

A cellular path shares no physical plant with the site’s wired carriers. Cisco recommends an LTE/5G link as “a secure, out-of-path management channel” for when “both in-band and out-of-band links are compromised”. Console servers with built-in modems use it for WAN failover and for OOB access when “WAN and LAN networks are down” (Opengear). But the modem also gives an internet-reachable path to a box that has console access to every device in the site, so treat it as a security boundary.

Why OOB matters more in colo

In a colo, your staff are usually not on site. The provider’s technicians do the physical work on request. Equinix Smart Hands covers tasks such as “power cycling”, “circuit testing”, and “on-site visual verifications”, and is meant for cases “where remote access is insufficient” (Equinix Smart Hands). CoreSite Remote Hands lists “remote serial console access to equipment”, “plug-in serial cables”, and “turn on, off or recycle device power” (CoreSite). Each task is a ticket, done by someone who does not know your topology, and some carry fees. For example, Equinix applies a Smart Hands fee when a cross connect it tests at your request turns out to be fine (Equinix cross connects). So design the common recoveries to need no ticket. Connect a console to every switch, router, and firewall. Make switched rack PDUs reachable over the OOB (PDU selection is on Power). Put server BMCs on the OOB switches. Give the OOB an uplink, either its own internet cross connect or cellular, that does not depend on the production carriers or edge.

Keep remote hands for hardware swaps and cabling. Onsite, the same design shortens recovery. In colo it is often the only fast path.

Console server shortlist

No analyst ranking (such as a Gartner Magic Quadrant) exists for console servers. This shortlist is an editorial ranking against six criteria, in priority order: (1) integrated 4G/5G modem, (2) central manager with call-home, (3) published security posture (TPM, secure boot, signed firmware, AAA with local fallback, ZTP), (4) dual A/B power in AC and DC, (5) 48 RS-232 per 1U plus USB console ports, and (6) public datasheet quality. DC positioning is used only to break near ties. Criterion 1 favors one appliance per pair of feeds. An external cellular router, placed where the signal is good and keeping the modem off the box that holds every console, is an equally valid design. Recheck the ranking in Q1 2027. Sources checked 6 Oct 2026.

Catalog → category Out-of-band management  ·  BOM Builder → add a console server to the OOB rack. Order the dual-AC or dual-DC SKU.

Runner-ups: Lantronix SLC 9000 is a near tie with Vertiv. It has TPM 2.0 on every SKU, 10G uplinks, and call-home ZTP, but its 5G modem is external and the product is new in 2026. SLC 8000 end-of-sale is 31 Dec 2026 (SLC 9000 product brief MPB-00346). The others are Digi Connect IT 16/48, with plug-in LTE Cat 4 and no published TPM or secure boot (Digi IT48), and Perle IOLAN SCG: no public datasheet found, and the store page omits RU, watts, TPM, and a central manager (Perle store). Last are async NIM modules on Cisco ISR 4000 routers, a router feature and not a dedicated appliance (Cisco async NIM data sheet, updated 2018). The data sheet lists ISR 4000 only. Support on Catalyst edge routers and lifecycle status are not verified.

OOB design / build / manage

Design

• Build physically separate OOB switching and routing. No management path should cross the production fabric, the production firewalls, or the production carriers.
• Use a dedicated management VRF and a non-overlapping address block, with no default route where feasible. Make AAA, syslog, NTP, and DNS reachable inside the OOB.
• Connect a console to every network device and firewall, and put BMCs, switched PDUs, rack ATS, and IT-rack UPS cards on the OOB
• Provide two OOB uplinks with no shared risk with production: wired, plus cellular on a private APN or VPN-only
• Fit dual power supplies on A and B feeds to OOB switches, routers, and console servers

Build

• Record the console and management port map, cross-referenced from the cabling as-builts on Racks
• Store break-glass local credentials where you can reach them during an outage, and test login with MFA unreachable
• With production uplinks pulled, reach every console, BMC, and PDU over each OOB path, including cellular
• Scan the cellular and OOB internet addresses from outside, and confirm that only the VPN or call-home path answers

Manage

• Alert on OOB device health, cellular signal, and SIM/plan status. Test the cellular path on a schedule.
• Run an OOB failover drill at least yearly
• Patch console servers and OOB switches on their own cadence. They can reach every device, which makes them high-value targets.
• Audit OOB and VPN accounts, and rotate break-glass credentials after every use
• Restrict outlet switching and UPS output control to named admin roles, and log every action. A UPS card can turn off every rack it feeds.
• Disable vendor cloud call-home on PDU and UPS cards, or allow it explicitly through OOB egress

Edge & carrier

This section covers where the data center meets carriers, the internet, cloud providers, and the DR site. Perimeter firewalls are on Security: Perimeter, DDoS protection on Security: DDoS, and SSE on Security: SSE / SASE. Branch WAN design (SD-WAN hub, carrier MPLS, hybrid) is in WAN design, and vendor shortlists in SD-WAN. Overlay design across sites is in EVPN/VXLAN: Multi-site and DCI. DCI links are in DCI to the DR site. Sources: IETF BCPs and RFCs, FCC/CSRIC, AWS, Microsoft, and Google documentation, and colo provider documentation. Cloud on-ramp SLAs, port speeds, and topology rules change often, so recheck them before you order. No pricing. Sources checked 5 Oct 2026.

Internet edge

The usual enterprise pattern is two edge routers, each connected to a different carrier, running eBGP to both, with the firewall pair behind them (placement and HA are on Security: Perimeter). Decide early whether you will announce your own ASN and provider-independent space or use carrier-assigned space. That choice controls how failover between carriers works. With your own space, both carriers announce the same prefixes and BGP moves inbound traffic when one fails. With carrier-assigned space, you can announce one carrier’s prefix through the other only with that carrier’s agreement, and you renumber if you leave it (RFC 4116 §3.1). Otherwise the site multihomes through NAT or per-host addresses from each carrier, and “transport-layer sessions will not, in general, survive over a re-homing event” (RFC 4116 §3.3). Inbound services then reach you only through the addresses published in DNS, so failing inbound traffic over between carriers is a DNS or NAT change, not a BGP one. BGP-diverted DDoS scrubbing also needs a prefix you can authorize the provider to announce (DDoS return path). RIR allocation takes time. BGP hygiene at the edge is standardized:

DDoS scrubbing: diversion and return path

The DDoS option is chosen on Security. The network work for BGP-diverted scrubbing:

• Prefix size. The provider announces your prefix, so it has to be a size the internet accepts. Cloudflare Magic Transit, as one example, requires “a minimum size of /24”. This ties back to the PI/ASN decision above.
• Authorization. Magic Transit requires a Letter of Agency for the prefixes and IRR entries that match the origin ASNs. Check the provider’s RPKI/ROA requirement too, so that ROV-validating networks accept its announcement.
• Return path MTU. Clean traffic usually comes back over GRE (or IPsec, or a direct interconnect). The GRE base header is 4 octets (RFC 2784). With an outer IPv4 header that is 24 bytes, so a 1500-byte path leaves a 1476-byte inner MTU and a 1436-byte TCP MSS. Cloudflare says to “Set a TCP MSS clamp to a maximum of 1,436 bytes” on edge transit ports for GRE ingress-only traffic. Clamp the MSS at the edge, because ICMP-based path MTU discovery is not reliable across the internet.
• Test. Bring the tunnels up and keep them monitored in normal service, and test diversion before you need it (Cloudflare Magic Transit).

Carrier diversity

“Two carriers” is a contract property, but diversity is a physical property. The FCC’s public-safety bureau, citing CSRIC best practice 12-9-0532, urges operators to “periodically audit the physical, logical, and provider diversity” of their networks “to ensure that a single outage will not simultaneously affect different circuits” (FCC DA 20-1406, Nov 2020). A May 2026 public bulletin that summarizes TIA-569-E and TIA-942-B says diverse routes belong in separate conduits, risers, and cable trays, on paths that do not share structural spaces or cross the same fire barriers. The same bulletin sets its own rules of at least 20 m between entrance points and no shared vaults (BC MoTT TE-2026-01). It cites the B revision. TIA-942-B has since been superseded by ANSI/TIA-942-C, published in May 2024 (TIA, 9 May 2024); see Facility: TIA-942. Entrance rooms and building entry points as spaces are on Facility. Ask each carrier for its route to the first shared point (building entrance, manhole, central office or POP, river crossing), and whether its last mile is leased from the other carrier. Re-audit after carrier changes, because grooming can collapse diversity without notice.

Meet-me room and cross connects (LOA/CFA)

Process. The Z-side party (a carrier, cloud provider, or partner) issues a Letter of Authorization. AWS and Google call it an LOA-CFA (Connecting Facility Assignment). It names the facility, panel, and port. You, as the A-side, submit it to the colo, which installs the cross connect. AWS: “If the cross connect is not completed within 90 days, the authority granted by the LOA-CFA expires” (AWS). Equinix accepts a digital LOA or an uploaded file, and takes the Z-side IBX from the LOA. On a trouble ticket, Equinix tests the circuit “from the A-side demarcation point to the Z-side demarcation point” (Equinix LOA ordering, Equinix cross connects). Digital Realty’s ServiceFabric guide follows the same pattern: download the LOA, then order the cross connect in the facility portal, with your equipment as the A-side (Digital Realty ServiceFabric).

Media and diversity. Equinix has offered only single-mode fibre for cross connects in its IBXs since 31 Mar 2024. Its Diverse Campus Cross Connects take a “guaranteed separate route” between buildings, with at least 20 ft of separation at the IBX manhole, but cross connects inside one IBX carry no such guarantee (Equinix). Other providers’ terms differ, so get the diversity commitment for redundant cross connects in writing. Keep LOAs, cross-connect IDs, and panel and port assignments here, cross-referenced from the cabling as-builts on Racks. The cage demarc panel as the plant boundary is on Racks: colo demarc.

Cloud on-ramps

All three use eBGP and expect redundant connections on separate customer devices. Size each link to carry the full load if its partner fails. Treat SLA tiers, port speeds, and location lists as fast-changing.

DCI to the DR site

Bandwidth. Size DCI from the backup and DR replication requirement, not from a guess. The RPO and the data change rate set it, and the method is on Backup: Replication bandwidth. Draft them before you order circuits, and add headroom for resynchronization after an outage.

Transport and encryption. Options range from dark fibre and carrier wavelengths to point-to-point carrier Ethernet, IP VPN, and internet IPsec or SD-WAN. AWS lists Metro Ethernet, dark fibre, and wavelength as typical L2 extensions (AWS). For L2 transports, MACsec (IEEE 802.1AE-2018) provides confidentiality, integrity, and origin authenticity per hop. On L3 transports, use IPsec. Per-packet overhead for both is in Encryption overhead on DCI. Confirm that the carrier service passes MACsec frames unchanged before you design around it. Apply the same physical-diversity audit as above to the two DCI paths. Overlay design across sites (RFC 9014 gateway models) is in EVPN/VXLAN: Multi-site and DCI. Size the DCI MTU for the replicated traffic plus any overlay and encryption overhead (MTU).

WAN design: SD-WAN hub, carrier MPLS, hybrid

This covers how branches and regional hubs reach the data center. The vendor shortlist and the DC hub vs branch CPE split are in SD-WAN. Remote-user access (SSE / SASE) is on Security. Sources checked 6 Oct 2026.

Hub placement and HA. Run two hubs per DC, off-path, each on every transport (Cisco: “connect to both transports for each WAN if possible”). Land the hubs’ LAN side in a WAN zone VRF at the border or service leaf, so branch traffic reaches the firewall the way Security’s policy model requires (Service insertion). Size the hubs for the total branch tunnel throughput, and size each hub to carry the full load alone. With a second DC, give it its own hub pair.

Edge & carrier design / build / manage

Design

• Use two carriers and two edge routers with eBGP to each. Decide on your own ASN and PI space versus carrier-assigned space early.
• Get each carrier’s route to the first shared point, and reject shared entrances, conduits, vaults, and central offices
• In colo, put redundant cross connects through separately routed MMRs and entrances where the facility has them, and get that commitment in writing
• Size DCI from the replication requirement on Backup. Size each cloud on-ramp link to carry the full load alone.
• Terminate BGP on routers and put the firewalls behind them. Firewall placement and HA are on Security.
• If DDoS scrubbing uses BGP diversion, plan the announced prefix size and the GRE return path, and clamp the MSS (DDoS return path)

Build

• Filter per BCP 194 (prefix, AS-path, max-prefix), apply BCP 38 egress filtering, and enable RPKI ROV
• Track LOA expiry dates. Take light readings on each cross connect and test end to end before turn-up.
• Fail each carrier, each edge router, each on-ramp link, and each DCI path under load
• Record circuit IDs, cross-connect IDs, LOAs, and demarc panel and port assignments, and cross-reference them in the cabling as-built (Racks)

Manage

• Re-audit physical diversity periodically and after any carrier change
• Monitor BGP sessions, prefix counts, ROV-invalid routes, and on-ramp utilization against failover headroom
• Track DCI utilization against replication lag, together with the backup team
• Recheck cloud on-ramp SLA and topology requirements at least yearly

Load balancers / ADC

Application delivery controllers (ADCs) are hardware, virtual, or software load balancers that publish VIPs for server pools. This section covers placement, TLS termination, sizing, and HA. Inbound decryption policy is on Security: Decryption, and firewall placement on Security: Perimeter. ADCs are a comparison category on the Stack shortlist. Vendor-agnostic. Product throughput figures are not comparable across vendors. Sources checked 5 Oct 2026.

Placement

In the fabric, attach hardware ADC pairs at the service leaf, in the same way as firewalls (Service insertion). Put each ADC interface in the zone VRF of the VIPs it serves: public VIPs in the DMZ behind the perimeter firewall, and internal VIPs in their application zone. One ADC pair with interfaces in several zone VRFs is an inter-zone path, so record it in Security’s inter-zone matrix (policy model). Only flows to a VIP pass the ADC. Do not route general east-west traffic through it.

TLS termination

There are three choices per VIP: terminate TLS at the ADC and send cleartext to the pool, terminate and re-encrypt to the pool, or pass TLS through at L4. Pass-through means no L7 features and no inspection. Wherever TLS terminates is where inbound inspection can happen, so the choice goes into Security’s decryption policy. Keys and certificates on the ADC fall under the same key management as the servers’.

Sizing

Size for the VIPs the ADC serves, north-south and internal, not for all east-west traffic. Handshake rate and bulk throughput are separate limits. RFC 9411 treats “TLS Handshake Rate”, “Inspected Throughput”, and connections per second as separate KPIs for security devices (RFC 9411). Apply the same split to ADCs. Short-lived HTTPS connections are usually handshake-bound, and certificate key type changes the handshake rate. Datasheets quote it per key type, so compare the key type you deploy. Add concurrent connections and SNAT port use. Size so one unit carries the full load after a failover.

HA

Active/standby with a floating VIP and config sync is the common baseline. Connection mirroring keeps long-lived sessions alive through a failover, but it costs capacity, so enable it per VIP where session loss matters. A failover moves the floating MAC between leaves, which the fabric sees as an EVPN MAC move. Test repeated failovers against duplicate-MAC detection (Service insertion). The HA trade-offs for firewalls on Security apply in the same way.

ADC design / build / manage

Design

• Choose two-arm or one-arm per pool. For one-arm, plan client-IP passing and SNAT pools.
• Place ADC interfaces in the zone VRF of their VIPs, and record any multi-zone ADC in the inter-zone matrix
• Set TLS termination per VIP with Security’s decryption policy

Build

• Load-test the handshake rate with your certificate key type, not only throughput
• Fail over under load several times in a row, and check session survival where mirroring is on

Manage

• Trend handshake rate, concurrent connections, and SNAT port use against the sizing
• Track certificate expiry per VIP
• Patch internet-facing ADCs on the emergency track (Security)

AI back-end (GPU-to-GPU) fabrics, including rail-optimized and multi-plane designs, RoCEv2 vs InfiniBand vs Ultra Ethernet, and AI fabric vendors, are on AI Factory: AI cluster networking.