View source for Extended disquisitions pertaining to eXpress data paths (XDP)

'''[[Dankblog|dankblog!]] 2023-04-20, 0423 EST, at [[Viewpoint|the danktower]]'''

I've spent the past three months building a substantial [[XDP]]-based application for my employer, intended to be run in [[Azure]] using the latter's "Accelerated Networking" ([[SR-IOV]]). By "substantial", I mean that it includes a significant userspace component using <tt>AF_XDP</tt> sockets, that the XDP code must use dynamic configuration data, and that it must operate on arbitrary hardware configurations. I've not seen any documentation that covers the details of such an application, whether official Linux kernel documentation, conference papers, or the essays of the technorati. Most examples involve simple packet filtering using static data, never touching on the <tt>AF_XDP</tt> funnel to userspace that makes XDP an actual eXpress Data Path and not just a good place to stick some [[eBPF]]. I hope with this post to somewhat remedy that situation.

These kinds of applications are typically developed atop the [[DPDK|Data Plane Development Kit]]. I chose XDP over the DPDK, and would be remiss were I not to first discuss that more established technology.

If you're new to XDP, you might want to read [[XDP|my page on that topic]]. I recently (2023-02-03) gave [[:File:Black - Fast Linux Networking 2023.pdf|a presentation]] to Microsoft Azure Orbital you might also read. I'm not a true expert on either of these two technologies yet, but I probably know more than you do.

==DPDK==
Intel's (now the Linux Foundation's) DPDK saw public release in late 2010 under the BSD license. It is mature technology with extensive hardware support and great documentation, and several general-purpose applications have been released on top of it. It consists of the Runtime Environment (RTE), the Environment Abstraction Layer (EAL), and Poll-Mode Drivers (PMDs) for various hardware and paravirtualized NICs. Devices are bound to Linux's [https://www.kernel.org/doc/html/latest/driver-api/vfio.html Virtual Function I/O] (VFIO) or [https://www.kernel.org/doc/html/latest/driver-api/uio-howto.html Userspace I/O] (UIO) subsystems rather than using their typical kernel drivers. This allows userspace to perform basic PCIe BAR configuration, necessary for registering buffers and enabling the card. The RTE sets up tx/rx data ("mbuf") rings (ideally backed by [[Pages|huge pages]]), and descriptor rings for same. The PMD then begins polling on the RX descriptor ring. Packets are received entirely without overhead, and processed in userspace without any compulsory context switches (mbufs can be transferred among cores using the "rings" subsystem of the RTE). Like any userspace networking system worth the price of entry, careful attention has been paid to isolation and affinity of threads, constraining allocation to local NUMA nodes, interrupt mapping, and NIC features like [[RSS]].

I'd argue that XDP is an implementation of [[Van Jacobson Channels|Van Jacobson channels]], while DPDK is a true userspace networking stack. The most fundamental difference is that a device bound to DPDK is <i>no longer a Linux networking interface</i> (usually. I believe there's something called dtap which gets around this somehow?). It will not show up in <tt>ip link</tt> output, and will not have an entry in <tt>/sys/class/net</tt> (it <i>will</i> show up on the PCIe bus). DPDK furthermore encompasses devices beyond network interface cards, including crypto, dma, and regex accelerators. The userspace application configures the device through UIO, and after that theoretically needn't touch the kernel (i.e. invoke system calls) to perform I/O. Everything is based off userspace ringbuffers and memory-mapped I/O.

With a dual-port 10GBASE-T Intel X550 (<tt>ixgbe</tt> driver), we see the following in <tt>lspci</tt>:
<pre>
44:00.0 Ethernet controller: Intel Corporation Ethernet Controller X550 (rev 01)
44:00.1 Ethernet controller: Intel Corporation Ethernet Controller X550 (rev 01)
</pre>
With no driver loaded, <tt>dpdk-devbind.py --status</tt> shows:
<pre>
Other Network devices
=====================
0000:44:00.0 'Ethernet Controller X550 1563'
0000:44:00.1 'Ethernet Controller X550 1563'
</pre>
We load the <tt>ixgbe</tt> driver, bringing up <tt>ixgbe1</tt>, and now see:
<pre>
Network devices using kernel driver
===================================
0000:44:00.0 'Ethernet Controller X550 1563' if=ixgbe0 drv=ixgbe unused=
0000:44:00.1 'Ethernet Controller X550 1563' if=ixgbe1 drv=ixgbe unused= *Active*
</pre>
We can unbind the driver from one of the ports using <tt>dpdk-devbind.py -u ixgbe0</tt>. Our <tt>ixgbe0</tt> interface disappears, and we now have:
<pre>
Network devices using kernel driver
===================================
0000:44:00.1 'Ethernet Controller X550 1563' if=ixgbe1 drv=ixgbe unused= *Active*

Other Network devices
=====================
0000:44:00.0 'Ethernet Controller X550 1563' unused=ixgbe
</pre>
We load the <tt>uio_pci_generic</tt>, <tt>vfio-pci</tt>, and <tt>igb_uio</tt> kernel modules (the first two are provided in kernel sources; the last comes from DPDK, and is built by DKMS):
<pre>
Network devices using kernel driver
===================================
0000:44:00.1 'Ethernet Controller X550 1563' if=ixgbe1 drv=ixgbe unused=igb_uio,vfio-pci,uio_pci_generic *Active*

Other Network devices
=====================
0000:44:00.0 'Ethernet Controller X550 1563' unused=ixgbe,igb_uio,vfio-pci,uio_pci_generic
</pre>
We now have one port suitable for use by a DPDK application using the <tt>igb_uio</tt> passthrough. Note that the UIO passthroughs are inferior to <tt>vfio-pci</tt> (which also would have worked), due to their inability to use [[IOMMU|IOMMUs]] and lack of DMA-safety. <tt>uio_pci_generic</tt> doesn't support even support [[MSI]], making it unusable under SR-IOV or with [[RSS]].

Anyway, this was supposed to be about <tt>AF_XDP</tt>. But there's your whirlwind introduction to DPDK. The essential bedrock fact is: <b>DPDK is a mature system for fast userspace packet dancing. It likes to poll. It cares about your machine and its setup only so far as it gets in DPDK's way.</b>

===Why AF_XDP?===
The fundamental value proposition of <tt>AF_XDP</tt> (aside from the fuzzy feeling of needing not leave the upstream kernel tree) is this: <b>DPDK wants to own its cards and their traffic.</b> By default, once a NIC is being used by DPDK, it's lost to the rest of the machine (this can be worked around using both software dispatch and NIC hardware queues, I believe). DPDK requires understanding a fairly large and esoteric system with its own <i>ex nihilo</i> API. The kernel receive path is rather more heavyweight than its transmit path; XDP focuses on this more usefully elided RX path. I expect both to fester/live through at least the decade's end.

====Before AF_XDP====
By the way, there are several other intersections of eBPF with the RX path for doing more or less this same kind of thing. You can attach eBPF to [https://docs.cilium.io/en/latest/bpf/progtypes/#tc-traffic-control ingress tc] (traffic control)'s <tt>cls_bpf</tt> classifier, with its own (incompatible) set of hooks and actions. This is what you want if you can accept the skbuff creation hit and want to work on that more flexible structure. It is furthermore largely device/driver-independent. You can also attach eBPF to a socket using the <tt>SO_ATTACH_BPF</tt> sockopt, though this can only AFAIK be used to filter ingress.

Rounding out this loose confederation is the esoteric <tt>SO_ATTACH_REUSEPORT_EBPF</tt>, which allows programmatic control of distribution among multiple listeners on the same <tt>REUSEPORT</tt>-aliased endpoint.

==XDP sweetness==
I'm not very interested in pure kernelspace XDP programs, and anyone who's spent quality time with the kernel's eBPF verifier is likely to agree. If you can get away with one for your task, awesome; do so. If all you need do is examine some headers, possibly scribble a little, maybe use a little per-cpu state, and drop the frame or kick it out some other interface, you need never leave the soft IRQ handler, and Godspeed. This is all reasonably well documented and straightforward.

Where I get really hot is <tt>AF_XDP</tt>, the (potentially zerocopy) path to userspace. In ways, it's like a fast <tt>AF_PACKET</tt> [[Packet_sockets|packet socket]] (on drivers without XDP acceleration, it's almost exactly like <tt>AF_PACKET</tt>), except that you can prevent the packet from progressing further through the kernel (using <tt>bpf_redirect_map()</tt> to divert to an <tt>AF_XDP</tt> socket inhibits further traversal whether the frame was copied or not, though you can clone the packet to facilitate traversal should you want to). Theoretically, we could get very close to DPDK's all-userspace, all-polled path: if we put our kernel code and userspace code on different processing elements (but accessing the same memories) we could very well match it (<b>NARRATOR: they would not match it.</b>).

Getting the best performance out of <tt>AF_XDP</tt> requires a lot of the same system administration-esque preconditioning needed by DPDK. We're coming up on two decades of mainstream multicore: you'll need to take advantage of parallelism via multiple threads. Those threads ought be bound to isolated processing elements, which means [[Control_Groups|cgroups]] and the [[Cpuset|cpuset]] controller. These days, that means [https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html CGroupsv2], ideally prepared in the form of a [[systemd]] [https://www.freedesktop.org/software/systemd/man/systemd.slice.html slice]. The standard technique is a "cpu shield": you pin your threads, one thread per core, and remove those cpus from other cgroups. Likewise, interrupt handlers (aside from our NIC queues) have our cores masked out of their affinities, while relevant interrupt handlers are bound to our cores (you can't generally move all unrelated kernel work away; some kernel tasks, including soft IRQs, are fundamentally per-cpu). In the language of Cgroupsv2, you'll want your cores in a root partition, or an isolated root partition if you don't need the crutch of a scheduling domain. On [[NUMA]] systems, create a partition per zone, and bind that group to the zone's local memory nodes, holding off on allocations until bound. Of course, cgroupsv2 and cgroupsv1 can kinda coexist in a fine example of Terrible Decisions by Smart People, so you might be on a cgroupsv2 machine that has old-school cpusets with their slightly but devastatingly different semantics, so know you're pretty much fucked I guess.

If you need serialization of a flow, direct it to a single queue, run your XDP on the single core to which that queue is directed, and stamp the frame there. You can then fling it to random processors. Otherwise, distribute frames among multiple XDP programs on different cores using multiple hardware queues. Queue counts can be retrieved with [[netlink]], but beyond that you're in ethtool country. Per-queue statistics are almost always available via ethtool's NIC-specific stats.

Use [[Pages|huge pages]] where appropriate. Intel's [[DDIO]] can be very effective. For physical NICs, ensure you're using cores in the same zone (usually this just means using cores from the package supplying the PCIe root to which the device is attached. Consult [[sysfs]] to get mappings at runtime). If you've got DDIO, you're on a Xeon and probably want to look at CAT (Cache Allocation Technology) as well.

Almost all of this stuff requires root or a swath of [[Linux_APIs#POSIX_capabilities|capabilities]], and is difficult to coordinate with other programs on arbitrary systems. Allah, the All-Powerful, has fucked us again!

===Configuring XDP programs===

Most existing XDP examples, especially of the kind one finds on cute little blog posts, are minimal and fairly useless. Oftentimes the author spends significant effort introducing the basics of eBPF, meaning you read more about [https://clang.llvm.org/ Clang] and [https://www.kernel.org/doc/html/latest/bpf/maps.html maps] than you do XDP. Once you finally bind some code, it's doing nothing more than checking ports or addresses against some compiled-in constants and dropping matches. Serious, production-level demonstrations are few and far between.

One element that's rarely touched upon is configuration of an XDP program, especially if that configuration is dynamic (i.e. it changes during runtime). The most basic means is of course embedding some constants in the C source. Leaving aside the annoyances of automatically modifying and non-interactively compiling code (let's not talk about directly patching binaries; it is after all peacetime), this is going to add some fundamental delay to your configuration cycle. The idea of rapidly reconfiguring via such a method can be rejected out of hand.

So the other obvious method is an eBPF map. Similarly to the process environment accessed with <tt>getenv(3)</tt> and <tt>setenv(3)</tt>, we can map up an array of <tt>__u32</tt> as so (in our XDP program):
<pre>
struct {                                                                                                                            
  __uint(type, BPF_MAP_TYPE_ARRAY);                                                                                                 
  __uint(max_entries, PARAM_COUNT);                                                                                                 
  __uint(map_flags, BPF_F_MMAPABLE);                                                                                                
  __type(key, __u32);                                                                                                               
  __type(value, __u32);                                                                                                             
} parameters SEC(".maps");
</pre>
then in our XDP-to-userspace interface definition (aka a header file) we might have, say:
<pre>
enum parameters {
  PARAM_IPV4_MATCH_COUNT,
  PARAM_IPV6_MATCH_COUNT,
  PARAM_THREAD_COUNT,
  PARAM_COUNT
};
</pre>
Userspace can set elements with <tt>bpf_map_update_elem()</tt>, and XDP can read them with <tt>bpf_map_lookup_elem()</tt>. These interfaces are atomic. It's a system call from userspace, which is lame, but not a big deal for write-once data structures or anything we're updating infrequently. From within eBPFspace, it's a [https://www.kernel.org/doc/html/latest/bpf/instruction-set.html#jump-instructions CALL] virtual instruction with no prologue/epilogue overhead, so essentially just an atomic read (all maps are of fixed size, and bounds checking is performed by the verifier). So some immediate questions:
* Q: is there atomicity if your value type is larger than a word?  A: as far as I can tell, this is implemented via [[RCU]].
* Q: can i cache the results? A: sure, <tt>static</tt> variables work as expected. but how will you invalidate this cache? and will it actually improve performance? how?
* Q: can my value type contain pointers? A: sure, but nothing (i hope obviously) chases-and-copies them or anything, and they're not immediately useful to you in kernelspace.
* Q: how do i make them useful? A: sigh, try <tt>bpf_probe_read_user_str()</tt> etc.
* Q: can the user safely modify an area i'm reading with <tt>bpf_probe_read_user_str()</tt>? A: no, that's how bitches die, obviously not.

==XDP problems==
There are some serious problems imho affecting XDP usability. Some of them seem mere technical challenges; some of them seem more fundamental.

===Overlap with desirable networking stack basics===
Back in 1998, Ptacek and Newsham wrote one of my all-time favorite infosec papers, "[https://insecure.org/stf/secnet_ids/secnet_ids.html Insertion, Deletion, and Denial of Service: Eluding Network Intrusion Detection]." I thought it a great paper then, and have thought about it regularly in the past twenty-five (jfc!) years. The central point is that an IDS/IPS can be only as confident about the traffic reaching a process as it can faithfully reproduce the networking between itself and that process. As a simple example, imagine an IPv4 packet is fragmented into two fragments A and B. Two different payloads for "A" are sent in succession using fragment offset 0, and only then is B sent. What does a userspace socket receive--the first "A" payload, or the second? It depends on the host's networking stack (and possibly on any middleware between the IPS and the host). If the IPS doesn't match (and know!) said stack (which is generally impossible), it can lead to errors.

XDP has some similar problems, and requires a good amount of boilerplate from the XDP program if all kinds of problems are not to be introduced. As the simplest example, a host will not normally deliver a payload to a userspace socket unless the packet was addressed to some address on the host. An XDP program forwarding to XSKs must perform this check itself (and there is no obvious way to access [[netlink]]-style stack details from an XDP program). <tt>rp_filter</tt> usually provides reverse path filtering as described in RFC 3704, but it is not applied to XDP. If a NIC is in promiscuous mode, frames will be delivered to XDP no matter their L2 destination address, etc.

* There seems no way to determine whether UDP and TCP checksums have been validated in hardware (there <i>are</i> methods for checksum deltas when modifying packets). If they have not, I usually want to validate them myself before passing packets to userspace, but if they have, I don't want to incur that cost. If a copy must be made of the packet data, I'd like any checksumming folded into that, where it will likely be hidden under memory access costs.

===Small annoyances===
* Binding an XDP program effectively cycles the device on some NICs.
* I can't determine what class of networking device I've been bound to from within my XDP program. Ought I expect Ethernet L2? Who knows!
* The <tt>rx_fill_ring_empty_descs</tt> statistic seems to come preloaded with a value of 1. lol?
* You can't generally take advantage of hardware offloading (LRO/TSO) in conjunction with XDP, though it seems you (sometimes) can with frag-aware XDP (read on...).
* How, oh how do I get hardware timestamps? <tt>SO_TIMESTAMP[NS]</tt> is sadly unsupported on <tt>AF_XDP</tt>.

===Mere technical issues===
* XSKs must bind to a specific NIC hardware queue. The APIs to get hardware queue information are old [[ethtool]] <tt>ioctl</tt>s, or the more recent netlink reimplementations of same. There are at least four types of hardware queue, dependent on hardware and driver. The APIs to *manage* hardware queues are barely there, yet to use XDP one absolutely must either (a) collapse all RX queues into a single queue or (b) direct all desired traffic to the XDP-bound queue with a NIC ntuple rule or (c) bind XDP programs to all RX queues. Per-queue statistics are completely NIC-dependent and unstructured with no unifying infrastructure despite near-universal support for basic per-queue stats. These are all really variations on the theme that "kernel-side ethtool seems to have been implemented by a drunken goldfish".

* If I've got a pool of workers, each with their own XSK, I'd often like to distribute packets to XSKs based on dynamic backlog, and I *never* want to dispatch to an XSK whose ringbuffer is full. The XSK abstraction offers no way to do this (you could build it yourself with eBPF maps).

* I ought be able to have all my rings backed by huge pages if I so desire, but this can't be accomplished with kernel-allocated rings as used with XSKs (the larger UMEM is allocated in user space, and can (should) use huge pages).

* Running libxdp-based programs under [[valgrind]] results in all kinds of illegal access errors being thrown, discomforting regarding such a low-level component. Also, if <tt>xdp-loader</tt> crashes while holding its filesystem-based lock, it can't be used again until the lock is deleted, even if the PID recorded therein no longer exists.

===Deeper problems===
* Though the infrastructure is there suggesting that multiple XDP applications can easily coexist on an interface, it's something of a charade. <tt>xdp-loader</tt> gives you a basic chaining infrastructure, sure, but take the checksum example above. In the absence of hardware checksums (and the ability to detect that they've been validated), must each XDP program validate? Must each XDP program check for L3 well-formedness? If they need defragmentation, must all programs carry that code? It seems to me that eBPF is sufficiently limited that the kernel ought be able to weave together various XDP programs, warn "this one will never match because its matches are all eaten by this already-loaded program", etc. Even that would probably be a mess. Until then, though, I think <tt>xdp-loader</tt>'s default policy of chaining is misguided.

* The jumbo frame support is a goddamn mess. Generally, you can't run XDP on an MTU larger than your page size less about a thousand bytes (3050 is typically as high as one can go on 4KiB pages). This effectively shoves an atomic bomb right up the ass of many high-speed networking setups. Do many people use Ethernet MTUs larger than 1500? No. Do many people who are interested in high-performance networking? Absolutely. Some drivers are rumored to support "multibuf"-based jumbo frames, but there's no easy way to discover whether you're working with such a driver. The best you can do seems to be setting the MTU high, setting the <tt>BPF_F_XDP_HAS_FRAGS</tt> flag on the <tt>struct bpf_program</tt>, and giving it the ol' college try. If it does load, it doesn't seem that <tt>XDP_OPTIONS_ZEROCOPY</tt> can be used, and your XDP program must use the "frag-aware API" if it's to access packet data in secondary fragments. Give me my 8K MTUs back!

* Locally-generated packets generally don't hit XDP unless you're binding to loopback. I.e. if I have an Intel X540 with an IP address of 172.16.88.40/24 at <tt>ixgbe0</tt>, and I bind an XDP program to it, and I ping 172.16.88.40 from that same machine, an XDP program bound to <tt>ixgbe0</tt> is <i>not</i> going to see that traffic, while XDP bound to <tt>lo</tt> <i>will</i>. This contradicts how local addresses typically work, requiring either that one explains the strange restriction to users (bonne chance!), bind to both devices (requiring two XDP programs and two sets of XSKs), or that one build an alternate path using regular sockets to handle this case.

* There doesn't seem to be any way to detect that you were unable to queue a packet to userspace via <tt>bpf_redirect_map()</tt> due to descriptor rings being full. If you're inducing reliability on a flow, this means an unfixable gap in your sequence numbers. Furthermore, it doesn't seem that such packets follow the third argument to <tt>bpf_redirect_map()</tt>, the policy on error. Instead, you just get a drop. If you could at least rely on <tt>XDP_PASS</tt> being honored, you could try to catch the packet on a regular socket.

==Executive summary==
XDP is mad decent, especially as DPDK gives me a terrible rash. With that said, it's by no means beautiful, and has a number of serious problems. It is easier to achieve peak rates with DPDK in most cases, <i>so long as you needn't use other kernel infrastructure</tt>, which is entirely bypassed.</i> Vendor DPDK sucks complete ass, because you <i>will</i> eventually need some very basic functionality that would normally be only an obscure [[iproute]] invocation away, whereas you will have to get your shitty vendor involved to customize their crap, and it will not be cheap. If you need transmit in addition to receive, use either XDP's TX functionality or [[io_uring]] with batched sends.

==One last thing==
DPDK lives on github at [https://github.com/DPDK/dpdk DPDK/dpdk] like every other reasonable project in existence, resting snugly in the loving bosom of Microsoft, somehow making us money (<i>disclosure: Microsoft has for several years now compensated me handsomely despite forthright sentiments like "Windows 11 is the worm-rotted cherry atop three decades of unceasing crapwaffle dogshit joke operating systems from Redmond. Anyone with half a brain in their head runs Linux, unless they run FreeBSD. Those unfortunate minutes of my day unwillingly spent in Outlook make me want to vomit. The other day there was a motherfucking movie advertisement in my "Start menu". Why anyone tolerates our garbage is a complete mystery to me, aside of course from the best-of-breed space/satellite service offerings due Microsoft Orbital. Kudos, Microsoft Orbital! Kudos to your handsome and stalwart maldevs, your gracious and skilled ladycoders, and your marvelous radomes! PS boss, if you're reading this, I need my laptop replaced yesterday."</i>).

When I sent a patch in to bring the XDP documentation up-to-date with the kernel's structure definitions, I was told to [https://patchwork.kernel.org/project/netdevbpf/patch/20230402084120.3041477-1-dankamongmen@gmail.com/ modify my git-diff] output and mail a different list. I mean, I get it, and I'm not going to tell you how to run your networking mailing list, but if this was github, that change would be in and I'd not be thinking about what's possibly broken about my <tt>git-diff</tt>. 

'''previously: "[[A_Rack_of_One%27s_Own|a rack of one's own]]" 2023-03-11'''

[[Category:Blog]]
[[Category:Networking]]