XDP: Difference between revisions

Line 76:

XSKs come in two varieties: <tt>XDP_SKB</tt> and <tt>XDP_DRV</tt>. <tt>XDP_DRV</tt> corresponds to Native Mode/Offload Mode XDP, and is faster than the generic <tt>XDP_SKB</tt>, but requires driver support. True zero-copy requires the use of <tt>XDP_DRV</tt> XSKs together with <tt>XDP_MODE_HW</tt> or <tt>XDP_MODE_NATIVE</tt> XDP programs.

How is <tt>AF_XDP</tt> different from [[Packet sockets|<tt>AF_PACKET</tt>]] (especially when the latter is backed with kernel-shared ringbuffers via <tt>PACKET_RX_RING</tt>)? Without mapped ringbuffers, packet sockets will always take an skbuff allocation (see diagram to the right) into which they will clone the packet, whereas XDP can avoid this outside of Generic Mode. Even with mapped ringbuffers, there is a ~~copy--the~~ packet socket fundamentally <i>clones</i> frames. XDP furthermore allows more direct control over parallelism, and can be tied into the NIC's hardware Receiver-Side Scaling. XDP allows RX and TX descriptors to point into the same memory pool, potentially avoiding copies. Note that the [https://lwn.net/Articles/737947/ AF_PACKETv4] proposal also used distinct descriptor rings.

How is <tt>AF_XDP</tt> different from [[Packet sockets|<tt>AF_PACKET</tt>]] (especially when the latter is backed with kernel-shared ringbuffers via <tt>PACKET_RX_RING</tt>)? Without mapped ringbuffers, packet sockets will always take an skbuff allocation (see diagram to the right) into which they will clone the packet, whereas XDP can avoid this outside of Generic Mode. Even with mapped ringbuffers, there is a copy—the packet socket fundamentally <i>clones</i> frames. XDP furthermore allows more direct control over parallelism, and can be tied into the NIC's hardware Receiver-Side Scaling. XDP allows RX and TX descriptors to point into the same memory pool, potentially avoiding copies. Note that the [https://lwn.net/Articles/737947/ AF_PACKETv4] proposal also used distinct descriptor rings.

Each XSK has either one or both of two RX and TX descriptor ringbuffers, along with a data ringbuffer (the UMEM) into which all descriptor rings point. The descriptor rings are registered with the kernel using <tt>XDP_RX_RING</tt>/<tt>XDP_UMEM_FILL_RING</tt> and <tt>XDP_TX_RING</tt>/<tt>XDP_UMEM_COMPLETION_RING</tt> with <tt>setsockopt(2)</tt>. UMEM is a virtually contiguous area divided into equally-sized chunks (chunk size is up to the user; it should usually be the MTU plus XDP overhead), and registered with the <tt>XDP_UMEM_REG</tt> sockopt. Packets will be written to distinct chunks. <tt>bind(2)</tt> then associates the XSK with a particular device and queueid, at which point traffic begins to flow. It is expected that chunks will be a multiple of the page size, but this does not appear to be necessary—in particular, success has been had doubling up two 2KB chunks on a 1500B MTU, despite amd64's minimum 4KB page size.

@@ Line 76: / Line 76: @@
 XSKs come in two varieties: <tt>XDP_SKB</tt> and <tt>XDP_DRV</tt>. <tt>XDP_DRV</tt> corresponds to Native Mode/Offload Mode XDP, and is faster than the generic <tt>XDP_SKB</tt>, but requires driver support. True zero-copy requires the use of <tt>XDP_DRV</tt> XSKs together with <tt>XDP_MODE_HW</tt> or <tt>XDP_MODE_NATIVE</tt> XDP programs.
-How is <tt>AF_XDP</tt> different from [[Packet sockets|<tt>AF_PACKET</tt>]] (especially when the latter is backed with kernel-shared ringbuffers via <tt>PACKET_RX_RING</tt>)? Without mapped ringbuffers, packet sockets will always take an skbuff allocation (see diagram to the right) into which they will clone the packet, whereas XDP can avoid this outside of Generic Mode. Even with mapped ringbuffers, there is a copy--the packet socket fundamentally <i>clones</i> frames. XDP furthermore allows more direct control over parallelism, and can be tied into the NIC's hardware Receiver-Side Scaling. XDP allows RX and TX descriptors to point into the same memory pool, potentially avoiding copies. Note that the [https://lwn.net/Articles/737947/ AF_PACKETv4] proposal also used distinct descriptor rings.
+How is <tt>AF_XDP</tt> different from [[Packet sockets|<tt>AF_PACKET</tt>]] (especially when the latter is backed with kernel-shared ringbuffers via <tt>PACKET_RX_RING</tt>)? Without mapped ringbuffers, packet sockets will always take an skbuff allocation (see diagram to the right) into which they will clone the packet, whereas XDP can avoid this outside of Generic Mode. Even with mapped ringbuffers, there is a copy—the packet socket fundamentally <i>clones</i> frames. XDP furthermore allows more direct control over parallelism, and can be tied into the NIC's hardware Receiver-Side Scaling. XDP allows RX and TX descriptors to point into the same memory pool, potentially avoiding copies. Note that the [https://lwn.net/Articles/737947/ AF_PACKETv4] proposal also used distinct descriptor rings.
 Each XSK has either one or both of two RX and TX descriptor ringbuffers, along with a data ringbuffer (the UMEM) into which all descriptor rings point. The descriptor rings are registered with the kernel using <tt>XDP_RX_RING</tt>/<tt>XDP_UMEM_FILL_RING</tt> and <tt>XDP_TX_RING</tt>/<tt>XDP_UMEM_COMPLETION_RING</tt> with <tt>setsockopt(2)</tt>. UMEM is a virtually contiguous area divided into equally-sized chunks (chunk size is up to the user; it should usually be the MTU plus XDP overhead), and registered with the <tt>XDP_UMEM_REG</tt> sockopt. Packets will be written to distinct chunks. <tt>bind(2)</tt> then associates the XSK with a particular device and queueid, at which point traffic begins to flow. It is expected that chunks will be a multiple of the page size, but this does not appear to be necessary—in particular, success has been had doubling up two 2KB chunks on a 1500B MTU, despite amd64's minimum 4KB page size.

Dank (talk | contribs)