dankwiki

Io uring: Difference between revisions

← Older edit

Dank (talk | contribs)

2 user groups
11,317 edits
 
(22 intermediate revisions by the same user not shown)
Line 11: Line 11:


==Rings==
==Rings==
Central to every uring are two ringbuffers holding CQEs (Completion Queue Entries) and SQE (Submission Queue Entries) descriptors (as best I can tell, this terminology was borrowed from the [https://nvmexpress.org/specifications/ NVMe specification]; I've also seen it with Mellanox NICs). Note that SQEs are allocated externally to the SQ descriptor ring (the same is not true for CQEs). SQEs roughly correspond to a single system call: they are tagged with an operation type, and filled in with the values that would traditionally be supplied as arguments to the appropriate function. Userspace is provided references to SQEs on the SQE ring, which it fills in and submits. Submission operates up through a specified SQE, and thus all SQEs before it in the ring must also be ready to go (this is likely the main reason why the SQ holds descriptors to an external ring of SQEs: you can acquire SQEs, and then submit them out of order). The kernel places results in the CQE ring. These rings are shared between kernel- and userspace. The rings must be distinct unless the kernel specifies the <tt>IORING_FEAT_SINGLE_MMAP</tt> feature (see below).
Central to every uring are two ringbuffers holding CQEs (Completion Queue Entries) and SQE (Submission Queue Entries) descriptors (as best I can tell, this terminology was borrowed from the [https://nvmexpress.org/specifications/ NVMe specification]; I've also seen it with Mellanox NICs). Note that SQEs are allocated externally to the SQ descriptor ring (the same is not true for CQEs). SQEs roughly correspond to a single system call: they are tagged with an operation type, and filled in with the values that would traditionally be supplied as arguments to the appropriate function. Userspace is provided references to SQEs on the SQE ring, which it fills in and submits. Submission operates up through a specified SQE, and thus all SQEs before it in the ring must also be ready to go (this is likely the main reason why the SQ holds descriptors to an external ring of SQEs: you can acquire SQEs, and then submit them out of order, but see <tt>IORING_SETUP_NO_SQARRAY</tt>). The kernel places results in the CQE ring. These rings are shared between kernel- and userspace. The rings must be distinct unless the kernel specifies the <tt>IORING_FEAT_SINGLE_MMAP</tt> feature (see below).


uring does not generally make use of <tt>errno</tt>. Synchronous functions return the negative error code as their result. Completion queue entries have the negated error code placed in their <tt>res</tt> fields.
uring does not generally make use of <tt>errno</tt>. Synchronous functions return the negative error code as their result. Completion queue entries have the negated error code placed in their <tt>res</tt> fields.
Line 57: Line 57:
|-
|-
| <tt>IORING_ENTER_REGISTERED_RING</tt> || <tt>ring_fd</tt> is an offset into the registered ring pool rather than a normal file descriptor.
| <tt>IORING_ENTER_REGISTERED_RING</tt> || <tt>ring_fd</tt> is an offset into the registered ring pool rather than a normal file descriptor.
|-
| <tt>IORING_ENTER_ABS_TIMER</tt> || (Since Linux 6.12) The timeout argument in the <tt>io_uring_getevents_arg</tt> is an absolute time, using the registered clock.
|-
|-
|}
|}
Line 217: Line 219:
|-
|-
| IORING_SETUP_NO_MMAP || 6.5 || The caller will provide two memory regions (the queue rings in <tt>cq_off.user_data</tt>, the SQEs in <tt>sq_off.user_data</tt>) rather than having them allocated by the kernel (they may both come from a single map). This allows for the use of huge pages.
| IORING_SETUP_NO_MMAP || 6.5 || The caller will provide two memory regions (the queue rings in <tt>cq_off.user_data</tt>, the SQEs in <tt>sq_off.user_data</tt>) rather than having them allocated by the kernel (they may both come from a single map). This allows for the use of huge pages.
|-
| IORING_SETUP_REGISTERED_FD_ONLY || 6.6 || Registers the ring fd for use with <tt>IORING_REGISTER_USE_REGISTERED_RING</tt>. Returns a registered fd index.
|-
| IORING_SETUP_NO_SQARRAY || 6.6 || Do not use an indirection array for the submission queue, instead submitting them strictly in order.
|-
|-
|}
|}
Line 258: Line 264:
|-
|-
| <tt>IORING_FEAT_REG_REG_RING</tt> || 6.3 || Ring descriptors can be registered with <tt>IORING_REGISTER_USE_REGISTERED_RING</tt>.
| <tt>IORING_FEAT_REG_REG_RING</tt> || 6.3 || Ring descriptors can be registered with <tt>IORING_REGISTER_USE_REGISTERED_RING</tt>.
|-
| <tt>IORING_FEAT_RECVSEND_BUNDLE</tt> || 6.10 || <tt>send</tt> and <tt>recv</tt> operations can be bundled.
|-
| <tt>IORING_FEAT_MIN_TIMEOUT</tt> || 6.12 || A minimum batch wait timeout is supported.
|-
|-
|}
|}
Line 357: Line 367:
|-
|-
| <tt>IORING_UNREGISTER_PERSONALITY</tt> || 5.6 ||
| <tt>IORING_UNREGISTER_PERSONALITY</tt> || 5.6 ||
|-
|}
====NAPI settings====
{| class="wikitable" border="1"
! Flag !! Kernel !! Description
|-
| <tt>IORING_REGISTER_NAPI</tt> ||  ||
|-
| <tt>IORING_UNREGISTER_PERSONALITY</tt> ||  ||
|-
|-
|}
|}
Line 408: Line 427:
|-
|-
| <tt>IORING_RECV_MULTISHOT</tt> ||
| <tt>IORING_RECV_MULTISHOT</tt> ||
|-
| <tt>IORING_SEND_MULTISHOT</tt> || Introduced in 6.8.
|-
|-
| <tt>IORING_ACCEPT_MULTISHOT</tt> ||
| <tt>IORING_ACCEPT_MULTISHOT</tt> ||
Line 554: Line 575:
// IORING_OP_CONNECT (5.5)
// IORING_OP_CONNECT (5.5)
void io_uring_prep_connect(struct io_uring_sqe *sqe, int sockfd, const struct sockaddr *addr, socklen_t addrlen);
void io_uring_prep_connect(struct io_uring_sqe *sqe, int sockfd, const struct sockaddr *addr, socklen_t addrlen);
</syntaxhighlight>
====Operations on file descriptors====
<syntaxhighlight lang="c">
// IORING_OP_URING_CMD
void io_uring_prep_cmd_sock(struct io_uring_sqe *sqe, int cmd_op, int fd, int level,
                            int optname, void *optval, int optlen);
// IORING_OP_GETXATTR
void io_uring_prep_getxattr(struct io_uring_sqe *sqe, const char *name, char *value,
                            const char *path, unsigned int len);
// IORING_OP_SETXATTR
void io_uring_prep_setxattr(struct io_uring_sqe *sqe, const char *name, const char *value,
                            const char *path, int flags, unsigned int len);
// IORING_OP_FGETXATTR
void io_uring_prep_fgetxattr(struct io_uring_sqe *sqe, int fd, const char *name,
                            char *value, unsigned int len);
// IORING_OP_FSETXATTR
void io_uring_prep_fsetxattr(struct io_uring_sqe *sqe, int fd, const char *name,
                            const char *value, int flags, unsigned int len);
</syntaxhighlight>
</syntaxhighlight>


Line 594: Line 634:
All transring messages are prepared using the target uring's file descriptor, and thus aren't particularly compatible with the higher-level liburing API (of course, you can just yank <tt>ring_fd</tt> out of <tt>struct iouring</tt>). That's unfortunate, because this functionality is very useful in a multithreaded environment. Note that the ring to which the SQE is submitted can itself be the target of the message!
All transring messages are prepared using the target uring's file descriptor, and thus aren't particularly compatible with the higher-level liburing API (of course, you can just yank <tt>ring_fd</tt> out of <tt>struct iouring</tt>). That's unfortunate, because this functionality is very useful in a multithreaded environment. Note that the ring to which the SQE is submitted can itself be the target of the message!
<syntaxhighlight lang="c">
<syntaxhighlight lang="c">
// IORING_OP_MSG_RING
void io_uring_prep_msg_ring(struct io_uring_sqe *sqe, int fd, unsigned int len, __u64 data, unsigned int flags);
void io_uring_prep_msg_ring(struct io_uring_sqe *sqe, int fd, unsigned int len, __u64 data, unsigned int flags);
void io_uring_prep_msg_ring_cqe_flags(struct io_uring_sqe *sqe, int fd, unsigned int len, __u64 data, unsigned int flags, unsigned int cqe_flags);
void io_uring_prep_msg_ring_cqe_flags(struct io_uring_sqe *sqe, int fd, unsigned int len, __u64 data, unsigned int flags, unsigned int cqe_flags);
void io_uring_prep_msg_ring_fd(struct io_uring_sqe *sqe, int fd, int source_fd, int target_fd, __u64 data, unsigned int flags);
void io_uring_prep_msg_ring_fd(struct io_uring_sqe *sqe, int fd, int source_fd, int target_fd, __u64 data, unsigned int flags);
void io_uring_prep_msg_ring_fd_alloc(struct io_uring_sqe *sqe, int fd, int source_fd, __u64 data, unsigned int flags);
void io_uring_prep_msg_ring_fd_alloc(struct io_uring_sqe *sqe, int fd, int source_fd, __u64 data, unsigned int flags);
</syntaxhighlight>
====Futexes====
<syntaxhighlight lang="c">
// IORING_OP_FUTEX_WAKE
void io_uring_prep_futex_wake(struct io_uring_sqe *sqe, uint32_t *futex, uint64_t val,
                              uint64_t mask, uint32_t futex_flags, unsigned int flags);
// IORING_OP_FUTEX_WAIT
void io_uring_prep_futex_wait(struct io_uring_sqe *sqe, uint32_t *futex, uint64_t val,
                              uint64_t mask, uint32_t futex_flags, unsigned int flags);
// IORING_OP_FUTEX_WAITV
void io_uring_prep_futex_waitv(struct io_uring_sqe *sqe, struct futex_waitv *futex,
                              uint32_t nr_futex, unsigned int flags);
</syntaxhighlight>
</syntaxhighlight>


Line 686: Line 740:
|-
|-
| <tt>IORING_CQE_F_NOTIF</tt> || Notification CQE for zero-copy sends
| <tt>IORING_CQE_F_NOTIF</tt> || Notification CQE for zero-copy sends
|-
| <tt>IORING_CQE_F_BUF_MORE</tt> || There will be more CQEs to the specified registered buffer
|-
|-
|}
|}
Line 774: Line 830:


Why is there no <tt>vmsplice(2)</tt> action? How about <tt>fork(2)</tt>/<tt>pthread_create(3)</tt> (or more generally <tt>clone(2)</tt>)? We could have coroutines with I/O.
Why is there no <tt>vmsplice(2)</tt> action? How about <tt>fork(2)</tt>/<tt>pthread_create(3)</tt> (or more generally <tt>clone(2)</tt>)? We could have coroutines with I/O.
It would be nice to have tight integration with condition variables or even mutex/futex (allow me to submit a request to get a lock, and when i get the CQE, i have that lock). Bonus points if the fast (uncontended) path never needs a system call (like mutexes built atop futexes today).
I understand that making huge pages as unpleasant as possible to use is a central tenet (perhaps <i>the</i> central tenet) underpinning the entire Linux mission, but it would be great if hugepage-backed urings were only a flag/field away (<b>UPDATE 2023-06-30:</b> IORING_SETUP_NO_MMAP has been [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=03d89a2de25b added for 6.5], huzzah!).


It's kind of annoying that chains can't extend over submissions. If I've got a lot of data I need delivered in order, it seems I'm limited to a single chain in-flight at a time, or else I risk out-of-order delivery due to one chain aborting, followed by items from a subsequent chain succeeding. <tt>recv_multishot</tt> can help with this.
It's kind of annoying that chains can't extend over submissions. If I've got a lot of data I need delivered in order, it seems I'm limited to a single chain in-flight at a time, or else I risk out-of-order delivery due to one chain aborting, followed by items from a subsequent chain succeeding. <tt>recv_multishot</tt> can help with this.
Line 783: Line 835:
It seems that the obvious next step would be providing small snippets of arbitrary computation to be run in kernelspace, linked with SQEs. Perhaps [[eBPF]] would be a good starting place.
It seems that the obvious next step would be providing small snippets of arbitrary computation to be run in kernelspace, linked with SQEs. Perhaps [[eBPF]] would be a good starting place.


It would be awesome if I could indicate in each SQE some other uring to which the CQE ought be delivered, basically combining inter-ring messages and SQE submission. This would be the <i>ne plus ultra</i> IMHO of multithreaded uring (and be very much like my [[Libtorque|masters thesis]], not that I claim any inspiration whatsoever for uring).
It would be awesome if I could indicate in each SQE some other uring to which the CQE ought be delivered, basically combining inter-ring messages and SQE submission. This would be the <i>ne plus ultra</i> IMHO of multithreaded uring (and be very much like my [[Libtorque|masters thesis]], not that I claim any inspiration/precedence whatsoever for uring).


==See also==
==See also==
Line 803: Line 855:
** "[https://lwn.net/Articles/863071/ Discriptorless files for io_uring]" 2021-07-19
** "[https://lwn.net/Articles/863071/ Discriptorless files for io_uring]" 2021-07-19
** "[https://lwn.net/Articles/879724/ Zero-copy network transmission with io_uring]" 2021-12-30
** "[https://lwn.net/Articles/879724/ Zero-copy network transmission with io_uring]" 2021-12-30
** "[https://lwn.net/Articles/1002371/ Process creation in io_uring]" 2024-12-20


[[CATEGORY: Networking]]
[[CATEGORY: Networking]]
Retrieved from "https://nick-black.com/dankwiki/index.php?title=Io_uring"