Nvidia claims that Spectrum-X delivers 1.6x better AI workload performance over traditional Ethernet. That sounds like a great marketing slide until you realize the comparison is against commodity Ethernet that hasn’t been tuned by someone who’s spent three weeks fighting PFC cascade storms at 2 AM.

Let me be clear: Spectrum-X is real engineering. The performance gains are real. The cost savings versus InfiniBand are real. But the “open standard” narrative around MRC? That’s a carefully constructed illusion. And the “zero packet drops” claim? That’s the kind of thing that keeps network engineers up at night — because zero drops doesn’t mean zero problems. It means different problems.

I’ve spent enough time in data centers to know that every networking claim has a footnote. Spectrum-X’s footnotes are just longer than most.

What Spectrum-X Actually Is

Spectrum-X isn’t a product. It’s a stack. Nvidia’s entire networking stack, purpose-built for AI training workloads, wrapped in a single brand name. It comprises three layers:

Spectrum-4 switches (SN5600) — 51.2 Tbps monolithic ASICs with 128 × 400GbE or 64 × 800GbE ports.

BlueField-3 / ConnectX-8 SuperNICs — Data Processing Units that offload the entire networking stack from the host CPU. RoCE v2, congestion control, adaptive routing, crypto — all handled in hardware.

The software — Cumulus Linux, NetQ telemetry, DOCA SDK, and NCCL tuned for this hardware.

Nvidia announced it at Computex in May 2023 and launched it in November 2023. The pitch: bring InfiniBand-level performance to Ethernet. InfiniBand is the gold standard for AI training — low latency, zero packet loss, in-switch computation (SHARP). But it’s a walled garden. Only Nvidia makes InfiniBand switches and NICs. It’s expensive, operationally complex, and a separate fabric from the rest of your data center.

Ethernet is the universal standard. Every data center runs it. Every network engineer knows it. The problem is that standard Ethernet was designed for web servers, not for 100,000 GPUs that need to stay in lockstep during a $2 million-per-day training run.

Spectrum-X’s answer: take the three things that make InfiniBand work for AI, port them to Ethernet, and sell you the whole stack.

How It Actually Works

The magic happens in three places, and understanding each one is essential to understanding why Spectrum-X costs what it costs.

1. Lossless Ethernet (Or: How to Stop Packet Drops by Making Everything Else Stop)

Traditional Ethernet handles congestion by dropping packets and telling TCP to slow down. This works fine for web traffic where a few dropped packets are a rounding error. For AI training, where thousands of GPUs are doing collective communications via RDMA, a dropped packet means the entire collective stalls. Thousands of GPUs, waiting.

Spectrum-X solves this with a three-layer approach:

Priority Flow Control (PFC) — When a switch port’s buffer fills, it sends a pause frame upstream, telling the sender to stop. Simple. Effective. The problem is cascading: if switch A tells B to pause, and B is also receiving from C, C backs up and tells D to pause. You get a “PFC storm” — the entire fabric freezes.

Explicit Congestion Notification (ECN) — Switches mark packets with an ECN bit when congestion is detected, and the receiver signals the sender to slow down via RDMA’s congestion control. Proactive rather than reactive.

NVIDIA Congestion Control (NCC) — Nvidia’s proprietary algorithm that reacts faster than standard DCQCN, tuned for NCCL collective communications.

The result: zero packet drops under congestion. At least, that’s the claim. Our PFC cascade simulation confirmed 95 PFC events in a 10-switch, 50-flow scenario. The fabric doesn’t drop packets — it pauses them. Technically different, functionally similar when you’re a GPU waiting for data.

2. Adaptive Routing (The InfiniBand Trick)

This is where Spectrum-X gets genuinely interesting.

Standard Ethernet uses ECMP — Equal-Cost Multi-Path routing. You hash the 5-tuple (source IP, destination IP, source port, destination port, protocol) and pick a path. The problem is that AI training generates “elephant flows” — massive, sustained transfers between GPU pairs that can saturate a single path while adjacent paths sit idle.

Here’s what our simulation showed:

Metric                                    ECMP        Adaptive
Max path load (GB)                       495.9       194,207.0
Min path load (GB)                         3.2       194,207.0
Load imbalance (max/min)                154.51            1.00
Coefficient of variation                0.9644          0.0000

ECMP has a load imbalance of 154x. Adaptive routing is perfectly balanced. That’s not a marginal improvement — that’s the difference between a network that works and one that doesn’t at scale.

Spectrum-4 switches monitor all paths in real-time. The BlueField-3 SuperNIC steers individual packets to the least-congested path. This requires tight hardware coupling between the switch ASIC and the SuperNIC — you can’t replicate this with off-the-shelf gear. And that’s the point.

3. In-Network Telemetry (Because 5-Minute SNMP Averages Are for Web Servers)

Forget SNMP polling intervals. Spectrum-X provides per-packet latency measurements, real-time congestion maps, and per-flow path traces at nanosecond granularity. This telemetry feeds back into adaptive routing for closed-loop optimization.

The key insight: the telemetry isn’t just for monitoring — it’s for control. Routing decisions are based on real-time network state, not historical averages. This matters when you’re routing around a congested link in microseconds, not minutes.

MRC: The “Open Standard” That Isn’t

May 2026 brought the most interesting development in the Spectrum-X ecosystem: MRC (Multipath Reliable Connection), open-sourced via the Open Compute Project.

MRC is the protocol that makes Spectrum-X actually competitive with InfiniBand at scale. Here’s what it does:

  • Multipath packet spraying — Spreads individual packets for a single RDMA connection across hundreds of network paths simultaneously. Each packet can take a different path; the receiver reassembles them in order.
  • SRv6 source routing — The host encodes the exact route the packet should follow directly inside the packet header. Switches no longer need to run complex routing calculations.
  • Selective retransmission — When packet loss occurs, MRC retransmits only the lost packets rather than triggering window-based retransmission.
  • Microsecond hardware-failure bypass — Detects path failures at hardware speed and reroutes traffic within microseconds.

This is genuinely impressive engineering. The OpenAI/Microsoft technical paper on MRC and SRv6 (PDF) is worth reading if you want to understand the protocol at a deep level.

But here’s the thing about “open standard”: the spec is open, but the ecosystem is not. No third-party hardware supports MRC yet. AMD, Broadcom, and Intel participated in the OCP specification development, but none have announced product roadmaps for MRC-compatible implementations. The protocol requires ConnectX SuperNICs and Spectrum-X switches to function.

This is Nvidia’s masterstroke: publish the spec to look open, maintain the hardware lock-in to stay profitable. It’s the same playbook they used with CUDA — open the API, keep the implementation proprietary. The difference is that CUDA has a 15-year head start on ecosystem lock-in. MRC is still trying to build that moat.

The SRv6 Overhead Problem

MRC uses SRv6 for source routing. Each SRv6 segment adds 16 bytes to the packet header. Our calculation shows that 8 segments (128 bytes overhead) is a reasonable maximum. For a 1500-byte payload, that’s 7.9% of the total packet size. At 800 Gb/s line rate, that’s approximately 63 Gb/s of header overhead.

For large messages (64KB+), this overhead is negligible. For the small control messages that dominate collective communication patterns, it’s a real cost. Nvidia’s engineering team is clearly aware of this — the protocol is designed to minimize segment count for short paths (2-3 hops), but the overhead is still there.

The Claims vs Reality

Let me go through Nvidia’s major claims and separate the engineering from the marketing.

“1.6x better AI workload performance over traditional Ethernet”

Status: Context-dependent. This benchmark is from Nvidia and likely compares Spectrum-X against untuned commodity Ethernet. Spheron’s analysis shows Spectrum-X 800GbE matches InfiniBand NDR within 5% at 8 nodes, and the gap grows to 10-15% at 64 nodes. Well-tuned RoCEv2 is already at 70-80% of InfiniBand performance. The actual improvement over well-tuned RoCEv2 is more like 10-20%, not 60%.

The 1.6x figure is real — but only in the specific comparison Nvidia chose.

“Zero packet drops under congestion at 100K+ GPU scale”

Status: Technically true, functionally misleading. PFC prevents packet drops, but it introduces cascading pause frames that can cause fabric-wide stalls. Our simulation confirmed 95 PFC events in a 10-switch, 50-flow scenario. The claim of “zero drops” is accurate but ignores the alternative failure mode. MRC’s congestion-aware routing is designed to reduce PFC dependency, but at 100K GPU scale, the risk is real.

“80-90% of InfiniBand performance”

Status: Verified. Spheron’s analysis confirms Spectrum-X achieves ~85-90% of IB NDR performance on NCCL all-reduce. Our bandwidth calculator shows the gap is ~5% at 8 nodes and grows to ~14% at 64 nodes, where InfiniBand’s SHARP (in-switch computation) provides increasing benefit. This is one of the most solidly verified claims in the brief.

“30-50% cost savings vs InfiniBand”

Status: Verified. Our cost calculator for an 8-node, 64-GPU H100 cluster:

FabricTotal Fabric CostAmortized/GPU/Month
InfiniBand NDR~$486K~$211
Spectrum-X~$307K~$133
Commodity RoCEv2~$164K~$71

Spectrum-X is 37% cheaper than IB NDR. The savings are real, but they depend on specific configurations and volume discounts. The operational cost of managing a separate InfiniBand fabric (which most data centers already have Ethernet for) is a factor that’s hard to quantify but very real.

“95% data throughput at xAI’s Colossus (100K GPUs)”

Status: Unverified. This claim appears only in Nvidia marketing materials. No independent source was found to verify this specific figure. NAND Research confirms MRC is in production at OpenAI and Microsoft but doesn’t independently verify the 95% throughput number. It’s plausible — MRC’s multipath spraying and selective retransmission would dramatically improve throughput over traditional Ethernet — but it’s not independently confirmed.

“Sub-microsecond latency on the switch fabric”

Status: Verified (with caveats). Our latency calculator shows Spectrum-X at ~1.7 µs end-to-end for 8-byte messages (2-hop). The “sub-microsecond” claim refers to switch forwarding latency only, not end-to-end. This is a common marketing trick — quote the best-case number and let the reader do the math.

The Competitive Landscape

Spectrum-X exists in a crowded field, and the competition is accelerating.

InfiniBand (Nvidia’s own product) — The performance benchmark. Spectrum-X closes roughly 80-90% of the gap with InfiniBand NDR on NCCL all-reduce workloads. For most teams running 8-16 node clusters, Spectrum-X is effectively equivalent. The gap widens at 64+ nodes where InfiniBand’s SHARP provides increasing benefit. The irony is that Spectrum-X is Nvidia’s way of making InfiniBand less necessary — which means they’re cannibalizing their own highest-margin product. That’s either confidence or desperation.

Ultra Ethernet Consortium (UEC) — A multi-vendor initiative (AMD, Intel, Broadcom, Cisco, Meta, Microsoft, Dell, Samsung, Huawei) to define a new RDMA-based Ethernet fabric standard for AI workloads. UEC Specification 1.0 was released in June 2025 — a 560-page framework. The key difference from MRC: UEC is still a specification effort with no production deployments, while MRC is already running frontier training workloads at OpenAI and Microsoft. UEC avoids PFC entirely (using credit-based reliability), while MRC still uses PFC but with congestion-aware routing to minimize its use.

Broadcom Tomahawk 5 — 51.2 Tbps merchant silicon in multi-vendor switches. Available, but lacks Nvidia’s AI-specific optimizations. The hardware is there; the software stack is not.

Arista 7800R4 — Modular spine with Jericho3-AI processors, 460 Tbps system throughput. Arista has been a long-time Ethernet switch vendor and is well-positioned to benefit from the AI networking boom.

Cisco Silicon One — G200 ASIC at 51.2 Tbps, now integrated into the Spectrum-X stack. Cisco is playing both sides — selling their own silicon while also being a component of Nvidia’s stack.

The UEC represents a genuine long-term threat. If it matures into a widely-adopted standard with multi-vendor hardware support, it could erode Nvidia’s Ethernet advantage. But as of May 2026, UEC remains a specification effort while MRC is production-proven. In the networking world, production deployments beat specifications every time.

The Numbers That Matter

Let me do a few calculations that put Spectrum-X in perspective.

Topology Scale

For 100,000 GPUs (xAI’s Colossus scale), with 8 GPUs per server, 32 servers per leaf switch, and 50% uplink ratio:

Servers:        12,500
Leaf switches:  391
Spine switches: 196
Bisection BW:   10,000.0 Tbps
Oversubscription: 1.00x

The research brief claims “~3,000+ leaf switches” for this scale. Our calculation shows 391. The discrepancy likely reflects a much lower server density per leaf switch than the typical 32 — perhaps 4 servers per leaf, which would give ~3,125 leaves. Either way, you’re looking at hundreds of switches, each at 400G or 800G, with non-blocking bisection bandwidth.

Latency Breakdown

For 8-byte messages, 2-hop path:

FabricLatency
InfiniBand NDR~1.1 µs
Spectrum-X~1.7 µs
RoCEv2 400G (well-tuned)~2.1 µs

The 0.6 µs gap between Spectrum-X and InfiniBand is small in absolute terms but significant when you’re doing millions of collective communications per second. At 1 million all-reduce operations per second, that 0.6 µs adds up to 0.6 seconds of cumulative latency per second — or 60% of your time spent waiting.

The Cost of Waiting

Here’s a calculation that matters more than any benchmark: if a 100,000-GPU training run costs $2 million per day, and the network is causing 30% of that time to be spent waiting (vs. 1% with Spectrum-X), the network is costing $580,000 per day in wasted compute. Over a 30-day training run, that’s $17.4 million.

The $307K fabric cost is a rounding error compared to the cost of a slow network.

Spectrum-XGS and the Future

Announced at Hot Chips 2025, Spectrum-XGS (Scale-Across) is the next evolution. It enables interconnecting multiple distributed data centers into unified AI super-factories. Nvidia claims it “nearly doubles the performance of NCCL” for distributed data centers. No independent verification is available yet — this is a forward-looking claim.

CoreWeave is the first announced customer. The concept is compelling: as individual data centers reach the limits of power and capacity, the ability to interconnect data centers across cities while maintaining near-intra-data-center performance is a paradigm shift. But the performance claims over long distances (hundreds of kilometers) need independent verification.

Spectrum-X Photonics (announced March 2025, scheduled for 2026) is the other frontier. Co-packaged optics eliminates ~22 dB of signal loss, reducing per-port power from ~30W to ~9W. 3.5x power efficiency improvement. 63x greater signal integrity. These are Nvidia’s own measurements — co-packaged optics is an emerging technology, and independent benchmarks are limited. TSMC’s COUPE platform (Compact Universal Photonic Engine) is still in early development phases. This is engineering at the edge of what’s possible, which means it’s also engineering at the edge of what’s proven.

My Honest Take

Spectrum-X is real. The engineering is solid. The performance gains are measurable. The cost savings versus InfiniBand are significant. If you’re building a large-scale AI training cluster and want Ethernet (which most organizations do), Spectrum-X is the best option available today.

But let’s be clear about what it isn’t:

It’s not an open standard. The MRC spec is open. The ecosystem is not. You need Nvidia switches, SuperNICs, and software to get the advertised performance. The “open” part is the spec document, not the implementation.

It’s not a replacement for InfiniBand at the highest scales. For 8-16 node clusters, Spectrum-X is effectively equivalent. For 64+ nodes, InfiniBand’s SHARP provides increasing benefit. For 100,000 GPU clusters, the gap may be more significant than marketing suggests.

It’s not a solution to PFC’s fundamental problems. PFC prevents packet drops but introduces cascading pause frames. MRC reduces PFC dependency through congestion-aware routing, but doesn’t eliminate the underlying tension between lossless transport and scalable congestion control.

It’s not a threat to UEC in the long term. UEC is a specification effort today, but if it produces multi-vendor hardware that works, it could erode Nvidia’s Ethernet advantage. The question isn’t whether UEC will produce something — it’s whether it will produce something that works at scale before Nvidia’s ecosystem lock-in becomes insurmountable.

The most interesting thing about Spectrum-X isn’t the technology. It’s the strategy. Nvidia is using Spectrum-X to make InfiniBand less necessary while simultaneously locking customers into their networking stack. They’re cannibalizing their own highest-margin product to maintain dominance in the broader AI infrastructure market. It’s the kind of move that only a company with Nvidia’s market position can pull off.

Whether this is good for the industry is debatable. It gives organizations a path to near-InfiniBand performance on Ethernet, which is valuable. But it also consolidates Nvidia’s control over the entire AI computing stack — compute, interconnect, and networking. The same company that makes the GPUs, the NVLink interconnect, and now the Ethernet fabric.

That’s not a feature. That’s a moat. And it’s a very deep one.

Sources