When a server in a GPU cluster dies, everyone knows within seconds. The failures that hurt more are the quiet ones: a link that still works but has become a bit slow, a NIC that keeps retransmitting. Nothing is down, no alarm fires, and the whole job slows down, because a collective such as an all-reduce finishes only when its slowest participant does. Disaggregated inference has the same weakness: the KV cache must cross the network between prefill and decode workers, and a slow path there is user-visible latency.

One slow link, and every GPU spends most of each step waiting.

These are gray failures, and I'm building AgentXDP, a per-node agent that measures how long RDMA communications take and points to the link that slows them down.

One limit up front: on real RDMA NICs, RoCE is processed in hardware and never reaches the host's network stack, so XDP sees nothing there. Production measurement has to come from NIC counters and from the communication libraries. XDP is for mirror ports, and for a test environment. This post is about that test environment.

Sensors feed a core library; results leave through Prometheus and OpenTelemetry. Dashed parts are planned.

Wall 1: Soft-RoCE won't move into a namespace

My VM has no RDMA hardware, so I use Soft-RoCE (rdma_rxe), which implements RoCE v2 in software on top of any Ethernet interface: real UDP 4791 packets, real transport headers, real ACKs. The plan was two network namespaces joined by a veth pair, one rxe device on each side.

Creating rxe from inside a namespace "worked", but the device was not there:

$ sudo ip netns exec ns-a rdma link add rxe-a type rxe netdev veth-a
$ sudo ip netns exec ns-a rdma link show rxe-a/1
Wrong device name
$ sudo rdma link show
link rxe-a/1 state ACTIVE physical_state LINK_UP

The network interface was in ns-a; the RDMA device had been registered in the initial namespace. Moving it explicitly failed:

$ sudo rdma dev set rxe-a netns ns-a
error: Operation not supported

The kernel source explains it (device.c, upstream v7.0): moving an RDMA device resets every client, including user-space contexts, so the core requires the driver to be able to disassociate them by force.

if (!dev->ops.disassociate_ucontext || ib_devices_shared_netns) {
        ret = -EOPNOTSUPP;

rxe doesn't implement that operation. Its UDP sockets are also created in the initial namespace (rxe_net.c, upstream v7.0), which ss confirms: one IPv4 and one IPv6 socket on port 4791, owned by no process, shared by every rxe device.

So both endpoints had to live in the initial namespace. The catch: both addresses are now local, and the kernel delivers 10.10.0.1 → 10.10.0.2 through loopback, invisible to XDP on the veth. Policy routing fixes that. Source-based rules, limited to locally generated packets (iif lo), send each address out of its own veth. The local table lookup moves below them, the new rule added before the old one is removed so that SSH never drops, and accept_local lets each side accept a packet whose source is local.

$ ip rule show
10:     from 10.10.0.1 to 10.10.0.2 iif lo lookup 101
11:     from 10.10.0.2 to 10.10.0.1 iif lo lookup 102
100:    from all lookup local
$ ip route get 10.10.0.2 from 10.10.0.1
10.10.0.2 from 10.10.0.1 dev veth-a table 101

Both Soft-RoCE endpoints in one namespace, with policy routing across the veth pair. The red INPUT gates are the subject of Wall 2.

Wall 2: the PASS that lied

My acceptance test was a capture: during a short ib_write_bw run, tcpdump on veth-b must show RoCE v2 packets. It passed. Meanwhile, ib_write_bw reported zero completed iterations.

Zero iterations, and a passing test.

$ sudo scripts/testbed.sh traffic bw -D 5
...
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      0           0.000000            0.000000            0.000000
$ sudo scripts/testbed.sh verify
...
07:16:35.642102 IP 10.10.0.1.49476 > 10.10.0.2.4791: UDP, length 4128
07:16:35.642158 IP 10.10.0.1.49476 > 10.10.0.2.4791: UDP, length 4112
...
== PASS: RoCE v2 crosses veth-b

The connection setup had worked, both sides had exchanged queue pair numbers and GIDs, and then nothing.

The device counters split the problem in two. The sender's rxe counted packets sent and then retry errors. The veth counted every packet on both ends, and the receiver's rxe counted nothing. The packets crossed the wire and died somewhere between veth-b and rxe.

Reading rxe's receive path showed four checks (size, destination GID, header and queue pair, CRC) that drop a packet silently. I had four suspects and no evidence, so I stopped guessing and asked the kernel where packets were being freed. Every dropped packet goes through kfree_skb, whose tracepoint carries the calling function and, on recent kernels, a drop reason.

Counting dropped packets by kernel function and drop reason.

$ sudo bpftrace -e 'tracepoint:skb:kfree_skb {
    @drops[ksym(args->location), args->reason] = count(); }'
Attaching 1 probe...


@drops[sk_stream_kill_queues, 106]: 2
@drops[nft_do_chain, 12]: 7611

Not rxe. nft_do_chain is netfilter, and reason 12, decoded from the tracepoint's format file, is NETFILTER_DROP:

$ sudo grep -o '{ 12, "[A-Z_]*" }' /sys/kernel/tracing/events/skb/kfree_skb/format
{ 12, "NETFILTER_DROP" }

The Oracle Cloud Ubuntu image ships an iptables INPUT chain that accepts established traffic, ICMP, loopback and SSH, and rejects everything else. My earlier ping went through the ICMP exception; perftest's TCP setup likely went through loopback. RoCE arriving on veth-b matched nothing and was rejected. The routing was fine all along.

tcpdump sits before the firewall: it saw every packet the firewall then dropped.

The fix is two rules, scoped to the test interfaces and never persisted, inserted by up and removed by down:

iptables -I INPUT 1 -i veth-a -j ACCEPT
iptables -I INPUT 1 -i veth-b -j ACCEPT

And the acceptance test changed: the capture is still required, but ib_write_bw must also complete at least one iteration. With both conditions, it passes, and down puts the firewall and the routing rules back exactly as they were.

The stricter test: packets on veth-b and completed RDMA writes.

What I learned

• An RDMA device and its network interface are separate objects, each with its own namespace membership. Check the right one.
• "Operation not supported" is an answer. Reading the kernel code took minutes; retrying flag combinations would have taken an evening.
• A capture proves arrival, not delivery. tcpdump sees packets at the interface, before the firewall and the socket. A test that checks only what you can see on the wire will happily pass a broken system.
• Ask the kernel who dropped the packet. One kfree_skb one-liner replaced four hypotheses. eBPF is the subject of this project, and the first useful eBPF program I ran on it was a debugging tool.
• Your environment has policies you didn't write. The firewall came with the image.

Next: an XDP program on veth-b that parses those RoCE v2 headers.