XDP: Difference between revisions

(6 intermediate revisions by the same user not shown)

Line 28:

* <tt>XDP_TX</tt>: emit the packet back out the interface on which it was received

* <tt>XDP_REDIRECT</tt>: send the packet to another NIC or an <tt>AF_XDP</tt> socket

Think of <tt>XDP_ABORTED</tt> as returning -1 from the eBPF program; it's meant to indicate an internal error. Theoretically, the packet could continue through the stack (as it would have if no XDP program had been running), but XDP uses "fail-close" semantics. Edits to the packet are carried through all XDP paths. Note that everything up through and including <tt>XDP_TX</tt> can be represented in two bits; this will be important later when passing flags to bpf helpers.

Think of <tt>XDP_ABORTED</tt> as returning -1 from the eBPF program; it's meant to indicate an internal error. Theoretically, the packet could continue through the stack (as it would have if no XDP program had been running, or <tt>XDP_PASS</tt> was returned), but XDP uses "fail-close" semantics. Edits to the packet are carried through all XDP paths. Note that everything up through and including <tt>XDP_TX</tt> can be represented in two bits; this will be important later when passing flags to bpf helpers.

<tt>xdp-loader status</tt> will list NICs and show details regarding any attached XDP programs. <tt>ethtool -S</tt> shows XDP statistics. <tt>ip link</tt> will also show a summary of devices' XDP chains. <tt>xdp_dispatcher</tt> is a metaprogram loaded by <tt>xdp-loader</tt> to support chaining multiple XDP programs; it will be present whenever programs were loaded using <tt>xdp-loader</tt>.

Line 76:

XSKs come in two varieties: <tt>XDP_SKB</tt> and <tt>XDP_DRV</tt>. <tt>XDP_DRV</tt> corresponds to Native Mode/Offload Mode XDP, and is faster than the generic <tt>XDP_SKB</tt>, but requires driver support. True zero-copy requires the use of <tt>XDP_DRV</tt> XSKs together with <tt>XDP_MODE_HW</tt> or <tt>XDP_MODE_NATIVE</tt> XDP programs.

How is <tt>AF_XDP</tt> different from [[Packet sockets|<tt>AF_PACKET</tt>]] (especially when the latter is backed with kernel-shared ringbuffers via <tt>PACKET_RX_RING</tt>)? Without mapped ringbuffers, packet sockets will always take an skbuff allocation (see diagram to the right) into which they will clone the packet, whereas XDP can avoid this outside of Generic Mode. Even with mapped ringbuffers, there is a ~~copy--the~~ packet socket fundamentally <i>clones</i> frames. XDP furthermore allows more direct control over parallelism, and can be tied into the NIC's hardware Receiver-Side Scaling. XDP allows RX and TX descriptors to point into the same memory pool, potentially avoiding copies. Note that the [https://lwn.net/Articles/737947/ AF_PACKETv4] proposal also used distinct descriptor rings.

How is <tt>AF_XDP</tt> different from [[Packet sockets|<tt>AF_PACKET</tt>]] (especially when the latter is backed with kernel-shared ringbuffers via <tt>PACKET_RX_RING</tt>)? Without mapped ringbuffers, packet sockets will always take an skbuff allocation (see diagram to the right) into which they will clone the packet, whereas XDP can avoid this outside of Generic Mode. Even with mapped ringbuffers, there is a copy—the packet socket fundamentally <i>clones</i> frames. XDP furthermore allows more direct control over parallelism, and can be tied into the NIC's hardware Receiver-Side Scaling. XDP allows RX and TX descriptors to point into the same memory pool, potentially avoiding copies. Note that the [https://lwn.net/Articles/737947/ AF_PACKETv4] proposal also used distinct descriptor rings.

Each XSK has either one or both of two RX and TX descriptor ringbuffers, along with a data ringbuffer (the UMEM) into which all descriptor rings point. The descriptor rings are registered with the kernel using <tt>XDP_RX_RING</tt>/<tt>XDP_UMEM_FILL_RING</tt> and <tt>XDP_TX_RING</tt>/<tt>XDP_UMEM_COMPLETION_RING</tt> with <tt>setsockopt(2)</tt>. UMEM is a virtually contiguous area divided into equally-sized chunks (chunk size is up to the user; it should usually be the MTU plus XDP overhead), and registered with the <tt>XDP_UMEM_REG</tt> sockopt. Packets will be written to distinct chunks. <tt>bind(2)</tt> then associates the XSK with a particular device and queueid, at which point traffic begins to flow. It is expected that chunks will be a multiple of the page size, but this does not appear to be necessary—in particular, success has been had doubling up two 2KB chunks on a 1500B MTU, despite amd64's minimum 4KB page size.

Line 168:

XDP requires that all the map entry counts be powers of two. The UMEM may not be more than 2**32 times the smallest page size; as this would be 16TiB on a 4KiB page, you probably needn't worry about that requirement.

The constants <tt>XSK_RING_CONS__DEFAULT_NUM_DESCS</tt> and <tt>XSK_RING_PROD__DEFAULT_NUM_DESCS</tt> (henceforth <tt>CONSDEF</tt> and <tt>PRODDEF</tt> respectively) are provided by libxdp; let's assume our consumer and producer rings respectively to have these as minimum sizes (if you'd like to hold more than ~~PRODDEF~~, just shift it however many positions to the left). If we have ''n'' threads, each with their own RX ring of at least <tt>~~PRODDEF~~</tt> entries, that implies backing by at least ''n * ~~PRODDEF~~'' UMEM entries (assuming equal distribution among threads). AFAIK there is no way to fill a packet and dispatch it to an RX ring later, so if our RX rings are full, we can't use any more than at most one extra fill descriptor. So our fill ring ought hold at least ''n * ~~PRODDEF~~'' descriptors, but the UMEM needn't support more than ''n * ~~PRODDEF~~ + 1'', and can reasonably get away with ''n * ~~PRODDEF~~''.

The constants <tt>XSK_RING_CONS__DEFAULT_NUM_DESCS</tt> and <tt>XSK_RING_PROD__DEFAULT_NUM_DESCS</tt> (henceforth <tt>CONSDEF</tt> and <tt>PRODDEF</tt> respectively) are provided by libxdp; let's assume our consumer and producer rings respectively to have these as minimum sizes (if you'd like to hold more than CONSDEF, just shift it however many positions to the left). If we have ''n'' threads, each with their own RX ring of at least <tt>CONSDEF</tt> entries, that implies backing by at least ''n * CONSDEF'' UMEM entries (assuming equal distribution among threads). AFAIK there is no way to fill a packet and dispatch it to an RX ring later, so if our RX rings are full, we can't use any more than at most one extra fill descriptor. So our fill ring ought hold at least ''n * CONSDEF'' descriptors, but the UMEM needn't support more than ''n * CONSDEF + 1'', and can reasonably get away with ''n * CONSDEF''.

Only one of these maps (the UMEM) is mapped in userspace. The descriptor rings are all mapped in the kernel, based off the size we provide. We probably want to use [[Pages|huge pages]] to back the UMEM (they're not currently available for the descriptor rings), and if so, it must be a multiple of some huge page size. Assuming amd64, 1GB is probably a bit much (256Ki 4KiB chunks), so that means a multiple of 2MiB (''is 1GB really too much? With an admittedly sky-high 800ms round trip time, 1GB is exactly one bandwidth-delay-product for 10Gbps...''). To minimize total allocation, then minimize waste, while satisfying our other constraints:

* ''UMEM_min'' is ''n * ~~PRODDEF~~ * chunksize'' bytes

* ''UMEM_min'' is ''n * CONSDEF * chunksize'' bytes

* ''UMEM_max'' is the minimum multiple of the desired huge page size such that ''UMEM_max >= UMEM_min''

* Set ''descs = ~~PRODDEF~~''

* Set ''descs = CONSDEF''

* While ''n * (descs << 1) * chunksize <= UMEM_max'', ''descs <<= 1''

''UMEM_max'' is the size of your UMEM in bytes, with support for ''n * descs'' chunks of ''chunksize'' bytes each. ''UMEM_max - n * descs * chunksize'' number of bytes wasted in this allocation. Each of 'n' RX rings supports ''descs'' descriptors (the actual size of the descriptor rings is somewhat larger than this would imply, and set by the kernel). The fill ring supports ''n * descs'' descriptors.

Fill ring descriptors are currently eight bytes, while RX ring descriptors are sixteen. On a 64-byte cacheline, it thus might make sense to fetch packets four at a time per thread, and perhaps prefetch subsequent packet data.

Fill ring descriptors are currently eight bytes, while RX ring descriptors are sixteen. On a 64-byte cacheline, it thus might make sense to fetch packets four at a time per thread, and perhaps prefetch subsequent packet data. Of course, this wouldn't be useful if your packet data is already resident in cache thanks to e.g. [[DDIO]].

==Driver details==

Line 194:

|-

| ixgbe (Intel) || Y || Y || Native || Y || Y

|-

| cxgb3 (Chelsio) || N || N || N || Y || N

|-

| mvneta (Marvell) || ? || Y || ? || Y || ?

Line 222:

Line 224:

* [[Io_uring|iouring]]

* [[100GbE]]

* "[[Extended_disquisitions_pertaining_to_eXpress_data_paths_(XDP)|eXtended Disquisitions Pertaining to eXpress Data Paths]]", a [[Dankblog|DANKBLOG]] post from 2023-04-20

==External links==

Line 228:

Line 231:

* [https://lwn.net/Articles/750845/ Accelerating networking with AF_XDP], LWN 2018-04-09

* [https://blog.packagecloud.io/monitoring-tuning-linux-networking-stack-receiving-data/ Monitoring and tuning the Linux Networking Stack: Receiving Data], PackageCloud 2016-06-21

[[CATEGORY: Networking]]

@@ Line 28: / Line 28: @@
 * <tt>XDP_TX</tt>: emit the packet back out the interface on which it was received
 * <tt>XDP_REDIRECT</tt>: send the packet to another NIC or an <tt>AF_XDP</tt> socket
-Think of <tt>XDP_ABORTED</tt> as returning -1 from the eBPF program; it's meant to indicate an internal error. Theoretically, the packet could continue through the stack (as it would have if no XDP program had been running), but XDP uses "fail-close" semantics. Edits to the packet are carried through all XDP paths. Note that everything up through and including <tt>XDP_TX</tt> can be represented in two bits; this will be important later when passing flags to bpf helpers.
+Think of <tt>XDP_ABORTED</tt> as returning -1 from the eBPF program; it's meant to indicate an internal error. Theoretically, the packet could continue through the stack (as it would have if no XDP program had been running, or <tt>XDP_PASS</tt> was returned), but XDP uses "fail-close" semantics. Edits to the packet are carried through all XDP paths. Note that everything up through and including <tt>XDP_TX</tt> can be represented in two bits; this will be important later when passing flags to bpf helpers.
 <tt>xdp-loader status</tt> will list NICs and show details regarding any attached XDP programs. <tt>ethtool -S</tt> shows XDP statistics. <tt>ip link</tt> will also show a summary of devices' XDP chains. <tt>xdp_dispatcher</tt> is a metaprogram loaded by <tt>xdp-loader</tt> to support chaining multiple XDP programs; it will be present whenever programs were loaded using <tt>xdp-loader</tt>.
@@ Line 76: / Line 76: @@
 XSKs come in two varieties: <tt>XDP_SKB</tt> and <tt>XDP_DRV</tt>. <tt>XDP_DRV</tt> corresponds to Native Mode/Offload Mode XDP, and is faster than the generic <tt>XDP_SKB</tt>, but requires driver support. True zero-copy requires the use of <tt>XDP_DRV</tt> XSKs together with <tt>XDP_MODE_HW</tt> or <tt>XDP_MODE_NATIVE</tt> XDP programs.
-How is <tt>AF_XDP</tt> different from [[Packet sockets|<tt>AF_PACKET</tt>]] (especially when the latter is backed with kernel-shared ringbuffers via <tt>PACKET_RX_RING</tt>)? Without mapped ringbuffers, packet sockets will always take an skbuff allocation (see diagram to the right) into which they will clone the packet, whereas XDP can avoid this outside of Generic Mode. Even with mapped ringbuffers, there is a copy--the packet socket fundamentally <i>clones</i> frames. XDP furthermore allows more direct control over parallelism, and can be tied into the NIC's hardware Receiver-Side Scaling. XDP allows RX and TX descriptors to point into the same memory pool, potentially avoiding copies. Note that the [https://lwn.net/Articles/737947/ AF_PACKETv4] proposal also used distinct descriptor rings.
+How is <tt>AF_XDP</tt> different from [[Packet sockets|<tt>AF_PACKET</tt>]] (especially when the latter is backed with kernel-shared ringbuffers via <tt>PACKET_RX_RING</tt>)? Without mapped ringbuffers, packet sockets will always take an skbuff allocation (see diagram to the right) into which they will clone the packet, whereas XDP can avoid this outside of Generic Mode. Even with mapped ringbuffers, there is a copy—the packet socket fundamentally <i>clones</i> frames. XDP furthermore allows more direct control over parallelism, and can be tied into the NIC's hardware Receiver-Side Scaling. XDP allows RX and TX descriptors to point into the same memory pool, potentially avoiding copies. Note that the [https://lwn.net/Articles/737947/ AF_PACKETv4] proposal also used distinct descriptor rings.
 Each XSK has either one or both of two RX and TX descriptor ringbuffers, along with a data ringbuffer (the UMEM) into which all descriptor rings point. The descriptor rings are registered with the kernel using <tt>XDP_RX_RING</tt>/<tt>XDP_UMEM_FILL_RING</tt> and <tt>XDP_TX_RING</tt>/<tt>XDP_UMEM_COMPLETION_RING</tt> with <tt>setsockopt(2)</tt>. UMEM is a virtually contiguous area divided into equally-sized chunks (chunk size is up to the user; it should usually be the MTU plus XDP overhead), and registered with the <tt>XDP_UMEM_REG</tt> sockopt. Packets will be written to distinct chunks. <tt>bind(2)</tt> then associates the XSK with a particular device and queueid, at which point traffic begins to flow. It is expected that chunks will be a multiple of the page size, but this does not appear to be necessary—in particular, success has been had doubling up two 2KB chunks on a 1500B MTU, despite amd64's minimum 4KB page size.
@@ Line 168: / Line 168: @@
 XDP requires that all the map entry counts be powers of two. The UMEM may not be more than 2**32 times the smallest page size; as this would be 16TiB on a 4KiB page, you probably needn't worry about that requirement.
-The constants <tt>XSK_RING_CONS__DEFAULT_NUM_DESCS</tt> and <tt>XSK_RING_PROD__DEFAULT_NUM_DESCS</tt> (henceforth <tt>CONSDEF</tt> and <tt>PRODDEF</tt> respectively) are provided by libxdp; let's assume our consumer and producer rings respectively to have these as minimum sizes (if you'd like to hold more than PRODDEF, just shift it however many positions to the left). If we have ''n'' threads, each with their own RX ring of at least <tt>PRODDEF</tt> entries, that implies backing by at least ''n * PRODDEF'' UMEM entries (assuming equal distribution among threads). AFAIK there is no way to fill a packet and dispatch it to an RX ring later, so if our RX rings are full, we can't use any more than at most one extra fill descriptor. So our fill ring ought hold at least ''n * PRODDEF'' descriptors, but the UMEM needn't support more than ''n * PRODDEF + 1'', and can reasonably get away with ''n * PRODDEF''.
+The constants <tt>XSK_RING_CONS__DEFAULT_NUM_DESCS</tt> and <tt>XSK_RING_PROD__DEFAULT_NUM_DESCS</tt> (henceforth <tt>CONSDEF</tt> and <tt>PRODDEF</tt> respectively) are provided by libxdp; let's assume our consumer and producer rings respectively to have these as minimum sizes (if you'd like to hold more than CONSDEF, just shift it however many positions to the left). If we have ''n'' threads, each with their own RX ring of at least <tt>CONSDEF</tt> entries, that implies backing by at least ''n * CONSDEF'' UMEM entries (assuming equal distribution among threads). AFAIK there is no way to fill a packet and dispatch it to an RX ring later, so if our RX rings are full, we can't use any more than at most one extra fill descriptor. So our fill ring ought hold at least ''n * CONSDEF'' descriptors, but the UMEM needn't support more than ''n * CONSDEF + 1'', and can reasonably get away with ''n * CONSDEF''.
 Only one of these maps (the UMEM) is mapped in userspace. The descriptor rings are all mapped in the kernel, based off the size we provide. We probably want to use [[Pages|huge pages]] to back the UMEM (they're not currently available for the descriptor rings), and if so, it must be a multiple of some huge page size. Assuming amd64, 1GB is probably a bit much (256Ki 4KiB chunks), so that means a multiple of 2MiB (''is 1GB really too much? With an admittedly sky-high 800ms round trip time, 1GB is exactly one bandwidth-delay-product for 10Gbps...''). To minimize total allocation, then minimize waste, while satisfying our other constraints:
-* ''UMEM_min'' is ''n * PRODDEF * chunksize'' bytes
+* ''UMEM_min'' is ''n * CONSDEF * chunksize'' bytes
 * ''UMEM_max'' is the minimum multiple of the desired huge page size such that ''UMEM_max >= UMEM_min''
-* Set ''descs = PRODDEF''
+* Set ''descs = CONSDEF''
 * While ''n * (descs << 1) * chunksize <= UMEM_max'', ''descs <<= 1''
 ''UMEM_max'' is the size of your UMEM in bytes, with support for ''n * descs'' chunks of ''chunksize'' bytes each. ''UMEM_max - n * descs * chunksize'' number of bytes wasted in this allocation. Each of 'n' RX rings supports ''descs'' descriptors (the actual size of the descriptor rings is somewhat larger than this would imply, and set by the kernel). The fill ring supports ''n * descs'' descriptors.
-Fill ring descriptors are currently eight bytes, while RX ring descriptors are sixteen. On a 64-byte cacheline, it thus might make sense to fetch packets four at a time per thread, and perhaps prefetch subsequent packet data.
+Fill ring descriptors are currently eight bytes, while RX ring descriptors are sixteen. On a 64-byte cacheline, it thus might make sense to fetch packets four at a time per thread, and perhaps prefetch subsequent packet data. Of course, this wouldn't be useful if your packet data is already resident in cache thanks to e.g. [[DDIO]].
 ==Driver details==
@@ Line 194: / Line 194: @@
 |-
 | ixgbe (Intel) || Y || Y || Native || Y || Y
+|-
+| cxgb3 (Chelsio) || N || N || N || Y || N
 |-
 | mvneta (Marvell) || ? || Y || ? || Y || ?
@@ Line 222: / Line 224: @@
 * [[Io_uring|iouring]]
 * [[100GbE]]
+* "[[Extended_disquisitions_pertaining_to_eXpress_data_paths_(XDP)|eXtended Disquisitions Pertaining to eXpress Data Paths]]", a [[Dankblog|DANKBLOG]] post from 2023-04-20
 ==External links==
@@ Line 228: / Line 231: @@
 * [https://lwn.net/Articles/750845/ Accelerating networking with AF_XDP], LWN 2018-04-09
 * [https://blog.packagecloud.io/monitoring-tuning-linux-networking-stack-receiving-data/ Monitoring and tuning the Linux Networking Stack: Receiving Data], PackageCloud 2016-06-21
+[[CATEGORY: Networking]]

Dank (talk | contribs)