PLNOG 32 - Routing IP w kontekście AI/ML i RoCEv2

### Overview This presentation examines backend AI/ML training clusters and explores how data center networking is shifting from custom fabrication toward mass production of standardized information processing substrates. The talk contrasts InfiniBand with Ethernet/IP running RDMA over Converged Ethernet version 2 (RoCEv2), evaluating how physical topologies, congestion control, and routing architectures influence the primary performance metric: Job Completion Time (JCT).

### Key Topics - **Backend Training Clusters and RDMA:** While frontend networks handle administrative duties, backend fabrics handle massive, long-lived, fat flows required for distributed linear algebra. RDMA directly maps memory between systems, bypassing traditional socket overhead (yielding roughly 4x speedups) and placing substantial stress on underlying network substrates. - **Network Topologies:** Topologies are evaluated by their ability to scale and handle congestion. Folded Clos networks remain the primary form factor (also utilized internally in Nvidia NVLink switches), but as clusters grow and face power constraints (~300 MW substation limits), alternate topologies like dragonfly, dragonfly+, and toroidal meshes (used in Google TPUs) emerge, each presenting distinct routing and recabling tradeoffs. - **InfiniBand vs. RoCEv2:** InfiniBand uses credit-based flow control, simple 4K packet formats, and centralized Subnet Managers, but faces addressing boundaries (limited to 48k addresses) and proprietary pricing pressures. RoCEv2 wraps InfiniBand in non-fragmentable UDP over IP/Ethernet, relying on DSCP/TOS priority mapping and ECN/PFC (802.1Qbb) for congestion management, though with challenges in entropy and packet misordering. - **Architectural Principles (CAP & Price of Anarchy):** Designing distributed routing systems involves balancing consistency, availability, and partition tolerance (CAP theorem). Centralized systems achieve resource efficiency under predictable workloads, whereas distributed systems provide rapid fault recovery and availability at the cost of efficiency (the Price of Anarchy).

### Technical Details - **Adaptive Routing and Out-of-Order Handling:** In standard RDMA fabrics, packet misordering triggers heavy retransmissions that degrade JCT. Vendor libraries such as Nvidia NCCL require specific extensions/hacks to safely handle packet misordering, allowing ECMP and multi-path routing to operate without retransmitting entire 1 GB blocks. - **Centralized vs. Hybrid IP Routing Approaches:** - *Hyperscale Hybrid Controller (Meta/Facebook model):* Uses BGP/EBGP as an underlying distributed routing base while deploying a centralized controller that interfaces with custom job schedulers and modified NCCL to inject specific longest-prefix-match exception routes, steering traffic away from congested paths. - *BGP Next-Next-Hop & Local Flooding:* Combines BGP with localized, one-hop flooding mechanisms to advertise real-time link congestion, allowing weighted ECMP decisions without relying on a centralized controller. - *IGP / RSVP-TE & FlexAlgo:* Running traditional link-state protocols with FlexAlgo enables per-prefix constraint computation, though full flooding and scaling overhead become bottlenecks at large cluster scales. - **Routing on Regular Topologies (RIFT):** Standard shortest-path routing fails on non-Clos fabrics like dragonfly+. Specialized protocols such as RIFT (Routing in Fat Trees) exploit fabric directionality by flooding topology information strictly northbound (southbound remains mostly default routing with selective prefix exceptions), providing loop-free, valley-free non-shortest-path routing at scale. - **In-Network Computing (SHARP):** Nvidia SHARP establishes in-network multicast trees across intermediate switches to perform mathematical reductions (such as floating-point summations) directly in hardware during packet transit, distributing final aggregated results downstream to worker nodes.

### Takeaways As AI clusters expand toward hundreds of thousands of accelerators, network design requires balancing the extreme efficiency of centralized scheduling against the rapid fault convergence of distributed IP routing. Emerging protocols like RIFT, alongside hybrid controller-based route injection and in-network computing substrates, represent the ongoing evolution to replace or augment specialized InfiniBand fabrics with scalable IP networks.