Hyperconverged infrastructure

Choosing HCI, dHCI or 3-tier

What each compute-platform model commits you to. Servers as hardware are on Compute; array and SAN design is on Storage. The rows are design judgment built on the sourced facts elsewhere on this page, except where a source is linked. Sources checked 6 Oct 2026.

Platform choice design / build / manage

Design

• Profile the workload’s compute-to-capacity ratio and growth; a skewed ratio favors dHCI or 3-tier
• Price licensing on the platform’s actual basis (cores, raw TiB), with a date on the quote
• Check the network demands of the shortlisted platform with the fabric team before choosing

Build

• Run a proof of concept on the target NICs and switches, including a node-loss rebuild

Manage

• Re-test the choice at each refresh; licensing and roadmaps move faster than hardware cycles

Cluster design

Node count, resiliency settings, failure domains, and stretched clusters with their witnesses. Platform figures are from vendor documentation; where a figure was not confirmed it is left out and flagged. No VMs-per-node ratio is given: none was found in a primary source, so the page gives a method. Sources checked 5–6 Oct 2026.

Sizing method

1. Take compute (vCPU, RAM) and usable capacity and performance from the workload inventory.
2. Apply the resiliency overhead of the chosen setting to get raw capacity (Resiliency). A three-way mirror leaves about one third of raw capacity usable (Microsoft Learn).
3. Keep free capacity to re-protect data after a node loss. Without it, a rebuild cannot complete and the cluster stays degraded.
4. Size so the remaining nodes carry the load with one node in maintenance (N+1). Add a second spare node (N+2) when a failure during maintenance must be tolerated (Microsoft Learn).
5. Split into clusters where it helps: management or infrastructure apart from workload, platform maximum cluster size, failure-domain size, and licensing boundaries.

Resiliency settings and node minimums

Failure domains

Decide which physical unit the cluster must survive losing, then map it to power feeds and switches. A rack that is a failure domain needs its own feeds (Power) and its own ToR pair. Nutanix block awareness needs at least 3 blocks and rack awareness at least 3 racks (Nutanix Bible). Azure Local takes the opposite default: all machines in a site sit in one rack on one ToR pair, and a rack-aware cluster is the supported way to span two racks (Microsoft Learn, rack aware clustering). Site loss needs a stretched or metro design, below.

Stretched, metro and rack-aware clusters, and witness placement

A stretched cluster spans two sites with synchronous writes, and a witness at a third site breaks ties when the sites lose contact. Put the witness where it does not share a failure domain with either data site. The network rules for these links are in Inter-site and witness links.

Cluster design / build / manage

Design

• Choose the resiliency setting first; it fixes the node minimum and the raw-to-usable ratio
• Size for N+1 (or N+2) after resiliency overhead, with free capacity for a full rebuild
• Map node, block, rack and site failure domains to power feeds and ToR pairs
• Place the witness at a third site that shares no failure domain with either data site

Build

• Confirm the platform reports the intended fault domains before loading data
• Fail a node, a rack (where designed for) and the inter-site link, and time the rebuild

Manage

• Re-check usable capacity against the rebuild reserve after every expansion
• Alert on any period of reduced redundancy, not only on outages

HCI network decisions

What the HCI platform demands of the network: traffic classes, host NIC speed, fit to the fabric, RDMA, switch requirements, switchless designs, and inter-site links. Shared definitions are not restated here: oversubscription is on Network: Fabric design, MTU and overlay overhead on Network: MTU, lossless Ethernet on Network: Lossless Ethernet, firewall insertion on Network: Service insertion, and the management network on Network: OOB. NIC hardware and host bonding are on Compute: Server networking. External-array protocols for dHCI are on Storage networking. Platform rules differ sharply, so they are compared rather than generalized. Sources checked 5–6 Oct 2026.

Cluster traffic classes

Host NIC speed and count

• vSAN ESA. VMware’s ReadyNode profiles run from 10 Gbps for the smallest (AF-0) through 25 Gbps (AF-2 to AF-6) to 100 Gbps (AF-8), and VMware calls 25 Gbps the practical minimum for new environments (VMware, Oct 2022).
• Azure Local. Each NIC must carry a Windows Server Catalog qualification for the traffic it carries (Management; Compute Standard or Premium; Storage Standard or Premium), and using an adapter outside its qualified type is not supported. Inbox drivers are not supported. Teaming is Switch Embedded Teaming only: switch-independent, up to eight identical adapters, no LBFO (Microsoft Learn). Microsoft’s converged example on 2 × 25 GbE reserves half the team for SMB, splitting it 70% storage, 29% live migration and 1% heartbeat. Microsoft labels this an example, not a support requirement.
• Nutanix. No primary NIC-speed guidance was fetched this pass.
• Dedicated or converged ports. VMware recommends dedicated ports for RDMA rather than mixing RDMA and TCP on one uplink (VMware, Jun 2025). Count ports per node for storage, VM and management, then match ToR port count to the cluster’s growth.
• No host LAG on RDMA storage ports. vSAN RDMA does not support LAG/LACP (Broadcom KB 382163), and Azure Local SET is switch-independent. The leaves can still be an MLAG or EVPN multihoming pair (Network: Multihoming), but the RDMA storage ports are single-homed, one per leaf, not bundled. Bonding mechanics: Compute: Server networking.

Fit to the fabric

The ratio math is on Network: Fabric design. VMware says that aiming for 1:1 oversubscription in a spine-leaf network gives vSAN the most consistent performance (VMware, May 2025). Microsoft asks you to minimize or preferably eliminate oversubscription (Microsoft Learn). The constraints below are specific to HCI platforms.

• Same rack, same ToR pair (Azure Local). All machines in a site must be in the same rack and on the same ToR switches (Microsoft Learn). This fixes rack and leaf planning (Racks): reserve the rack, power and ToR ports for the cluster’s full size.
• One subnet and VLAN per storage NIC side (Azure Local). With both storage ports on one subnet, SMB Multichannel opens cross connections that congest the MC-LAG peer link. Microsoft recommends a separate subnet and VLAN per side so left-side traffic stays on ToR 1 and right-side on ToR 2 (Microsoft Learn).
• Storage traffic stays in Layer 2. Azure Local storage is not routable (Microsoft Learn). Broadcom says not to run vSAN RoCE over Layer 3 or use LAG/LACP with RDMA (Broadcom KB 382163), so the vSAN RDMA interfaces share one L2 segment. An L2 VNI stretched across leaves is still L2, so this is a design choice, not a rule: spanning racks means carrying lossless Ethernet over VXLAN, which is platform-dependent (Network: Lossless Ethernet), while confining the RDMA cluster to one leaf pair is simpler and is the recommended design. vSAN over TCP has no such constraint.
• Same active uplink on every host. Set an explicit failover order so every host uses the same active storage uplink. Storage then stays on one leaf and off the peer link.
• Consistent L2 settings (vSAN). Every vSAN interface in a cluster uses the same VLAN, MTU and subnet; verify with vmkping at the configured MTU (Broadcom techdocs).
• MTU. Set the storage interface MTU end to end and prove it with DF-set pings at that size. Overlay overhead and underlay MTU are on Network: MTU. Azure Local SDN compute needs a switch MTU between 1514 and 9174 (Microsoft Learn).
• Buffers (vSAN). VMware looks for at least 1 MB of buffer per port on switches without VoQ, and notes buffers do not make up for link speed (VMware, Oct 2022).
• Hop count (Nutanix). Nutanix guidance is reported as leaf-spine with no more than three switches between nodes in a cluster.

RDMA for HCI

RDMA cuts host CPU and latency for storage traffic, but RoCE puts lossless-Ethernet configuration on every switch port in the path. Decide per platform, and only where the network team will run DCB. PFC, ETS, ECN and DCQCN mechanics, and the PFC watchdog, are explained on Network: Lossless Ethernet.

Vendor switch requirements

Sources: Azure Local physical network requirements (ms.date 27 Mar 2026); vSAN network checklist; Broadcom KB 382163. Check the switch OS version against the platform release before every switch or cluster upgrade.

Switchless (direct-connect) small clusters

Azure Local can run storage over direct full-mesh links between machines, with redundant links per pair. ToR switches are still needed for management and VM traffic. Direct links remove the switch QoS, PFC and ECN configuration, but Network ATC still configures the host storage intent, and the RDMA protocol, adapters and firmware must match the validated pattern. Host-side DCB still applies to RoCE on the direct links; only the switch policy drops out. Storage-switchless supports at most 4 machines, and Microsoft recommends 3 for production. The benefit falls off above 3 because of NIC count, and adding a machine means moving to switched storage or redeploying (Microsoft Learn, physical network). vSAN also has a 2-node direct-connect design. Choose switchless only for clusters that will not grow.

Inter-site and witness links

Latency, bandwidth and witness budgets per platform are in the stretched cluster table. The overlay between sites is designed on Network: Multi-site and the DCI circuits on Network: DCI.

• vSAN witness traffic is routed (Layer 3) from each data site to the witness, even when the two data sites share Layer 2 (vSAN Stretched Cluster Guide).
• With witness traffic separation, the witness path can use a different MTU from the inter-site link, for example 9000 between data sites and 1500 to the witness (same source). The 9000-byte data MTU must still fit the DCI after overlay and encryption overhead (Network: MTU).
• Keep firewalls, NAT, IDS/IPS and WAN optimizers out of the vSAN VMkernel path (same source). Carry this traffic in a VRF with no firewall in the path, and record it as L3/L4-only in the inter-zone matrix of the segmentation policy model.
• Size asynchronous replication links from RPO and change rate on Backup: Replication bandwidth.

HCI network design / build / manage

Design

• Decide RDMA or TCP per platform, and whether the network team will run DCB on the storage ToR ports
• Pick NIC speed and count against the platform minimum and rebuild traffic, not steady-state load
• Place each cluster per the platform’s rack and ToR rules, and reserve ports for its full size
• Give each storage NIC side its own subnet and VLAN where the platform recommends it
• Record the DCB priorities and ETS shares with the fabric team so hosts and switches match
• Budget inter-site and witness links before choosing a stretched design

Build

• Apply the platform’s switch reference configuration (PFC, ETS, LLDP TLVs, VLANs) to storage ToR ports
• Prove MTU end to end with DF-set pings at the configured size on every storage path
• Confirm RDMA is actually in use under load (host and switch counters), not only that pings pass
• Pull a link and a node under load; watch resync and confirm storage traffic stays off the ToR peer link

Manage

• Watch PFC pause frames, PFC watchdog events, ECN marks and RDMA retransmits
• Check the switch OS against the platform’s list before switch upgrades
• Re-validate RDMA and MTU after NIC firmware or driver updates
• Trend ToR uplink use during resync and maintenance windows

Lifecycle

Compatibility lists, orchestrated updates and expansion rules. Release cadences and supported node generations change often; recheck at every revision. Sources checked 5–6 Oct 2026.

Compatibility lists

Orchestrated updates

HCI platforms update the full stack (BMC and drive firmware, NIC drivers, hypervisor, storage software) against a validated bundle, rolling one node at a time while data is evacuated or resynced. Examples are vSphere Lifecycle Manager, Nutanix LCM, Azure Local solution updates and VxRail Manager. Each node out of service is a capacity and redundancy reduction, so updates need the same headroom as a node loss. vSAN can upgrade the witness appliance in the same vLCM workflow as the cluster (vSAN Stretched Cluster Guide). Azure Local documentation is versioned by monthly release (for example 2603 to 2610), so check requirements against the release you run (Microsoft Learn).

Expansion

• Expand by node within the platform’s list and generation rules; mixed generations may be limited
• Where capacity is licensed on raw TiB, adding drives changes the subscription (Licensing basis)
• Azure Local switchless clusters cannot add a machine in place; rack-aware clusters grow in pairs, 2 + 2 to 3 + 3 to 4 + 4 (switchless, rack aware)
• Under a same-rack rule, the rack, power and ToR ports for growth must be reserved at day one

Lifecycle design / build / manage

Design

• Pick nodes and NICs from the current list, and check how long that generation stays supported
• Plan expansion steps and the rack, power and switch ports they need

Build

• Bring every node to the same validated bundle before the cluster is created

Manage

• Patch on a controlled cadence through the orchestrated path, never node by node by hand
• Track node end of life, drive endurance and the list status of each component

HCI DR: built-in replication

What the HCI platform provides natively, and where it stops. Backup platforms, immutability, DR runbooks and replication bandwidth sizing are on Backup. Sources checked 5 Oct 2026.

Replication options

Replication is not backup

Snapshots and replication share the cluster’s failure and admin domain. Corruption, ransomware or a bad admin action replicates as faithfully as good data, and an attacker with cluster admin rights can delete both copies. Keep backup to an independent, immutable target as well (Backup). Map each workload’s RPO and RTO to a mechanism: synchronous for zero RPO within metro distance, asynchronous for the rest, and backup for recovery from logical damage.

HCI DR design / build / manage

Design

• Assign each workload tier a mechanism from its RPO and RTO; leave link sizing to Backup
• Keep backup admin identities and targets outside the HCI admin domain (Identity)

Build

• Hand HCI snapshots and replication off to the backup platform; test a restore, not only a failover

Manage

• Alert on replication lag against RPO; rehearse failover and failback on a schedule