XDP/BPF in RDMA(RXE)

## Overview

This talk presents an implementation of eXpress Data Path (XDP) integrated with Remote Direct Memory Access (RDMA) in SoftRoCE (software RDMA over Converged Ethernet). The speaker demonstrates how to combine XDP's high-performance packet processing capabilities with RDMA protocol handling for efficient network data inspection and forwarding.

## Key Topics

**XDP (eXpress Data Path)** XDP is a Linux kernel feature introduced in version 4.8 that provides high-performance packet processing at the earliest point in the network stack, close to hardware. It uses eBPF (extended Berkeley Packet Filter) bytecode for portable, platform-independent packet processing across various platforms including Linux, network switches, and DPDK. Primary use cases include: dropping malicious packets to reduce CPU load, load balancing via packet redirection, packet sampling and monitoring for content inspection, and fast forwarding to other network interfaces or user space applications. XDP defines five return actions: ABORTED, DROP (silently discards packets), PASS (forwards to network stack), TX (transmits back to interface), and REDIRECT (redirects to other interfaces or user space).

**RDMA (Remote Direct Memory Access)** RDMA is a protocol for high-speed network interconnection that overcomes performance barriers inherent in traditional networking stacks like TCP/IP and Ethernet. Two main implementations are discussed: InfiniBand (requires specialized hardware including Host Channel Adapters, IB switches, and special network interface cards) and RoCE (RDMA over Converged Ethernet). RoCE version 1 operates at the TCP/IP layer 2, works only in local area networks, and is unpopular in data centers. RoCE version 2, based on UDP/IP with UDP port 4791, is widely adopted in public and private cloud environments. SoftRoCE is a software implementation of the RoCE v2 specification that operates over standard Ethernet networks, making it accessible for developers without specialized hardware.

**Architecture and Implementation** The system comprises three components: user space (SoftRoCE user driver and eBPF APIs), kernel space (SoftRoCE kernel driver attached to NIC drivers like Intel E810), and hardware layer. The high-level design flow shows the NIC driver transferring XDP programs to the SoftRoCE driver, which maps RDMA data between kernel and user space. A new XDP flag (XDP_FLAG_RDMA) was introduced to enable this functionality. The implementation includes C structures for packet storage and analysis. In the kernel, the NIC driver checks the XDP_FLAG_RDMA flag; if set, the XDP program transfers to SoftRoCE driver; if not, the original XDP setup is used. The SoftRoCE driver handles XDP initialization considering both scenarios: when the driver is attached before user space applications and when applications are already running.

**Send and Receive Path Processing** On the send path, RDMA data from the stack is built into an SKB package, sent through UDP payload, and simultaneously cloned to the XDP packet queue for analysis by XDP programs. On the receive path, packets arriving at the NIC are passed through the XDP program with UDP port 4791 filtering; if XDP is enabled, the XDP program processes the cloned SKB package before user space applications receive it. The original packet continues through the RDMA stack in both cases.

**Demo and Applications** The speaker demonstrated a four-window display capturing and analyzing RoCE protocol packets from both IP server and client sides. The primary application is sampling RDMA packet content for validation in data center environments, helping developers verify RoCE v2 protocol parameter correctness and debug RDMA application implementations. The tool is already in use by several companies and is intended for upstream kernel contribution.

## Takeaways

- XDP combined with RDMA enables efficient packet inspection at the kernel's lowest network layer while maintaining normal stack processing - SoftRoCE democratizes RDMA capabilities by providing software implementation over standard Ethernet - The implementation is production-ready with multiple companies already deploying it - Kernel changes are planned for upstream submission