Io uring

(19 intermediate revisions by the same user not shown)

Line 11:

==Rings==

Central to every uring are two ringbuffers holding CQEs (Completion Queue Entries) and SQE (Submission Queue Entries) descriptors (as best I can tell, this terminology was borrowed from the [https://nvmexpress.org/specifications/ NVMe specification]; I've also seen it with Mellanox NICs). Note that SQEs are allocated externally to the SQ descriptor ring (the same is not true for CQEs). SQEs roughly correspond to a single system call: they are tagged with an operation type, and filled in with the values that would traditionally be supplied as arguments to the appropriate function. Userspace is provided references to SQEs on the SQE ring, which it fills in and submits. Submission operates up through a specified SQE, and thus all SQEs before it in the ring must also be ready to go (this is likely the main reason why the SQ holds descriptors to an external ring of SQEs: you can acquire SQEs, and then submit them out of order). The kernel places results in the CQE ring. These rings are shared between kernel- and userspace. The rings must be distinct unless the kernel specifies the <tt>IORING_FEAT_SINGLE_MMAP</tt> feature (see below).

Central to every uring are two ringbuffers holding CQEs (Completion Queue Entries) and SQE (Submission Queue Entries) descriptors (as best I can tell, this terminology was borrowed from the [https://nvmexpress.org/specifications/ NVMe specification]; I've also seen it with Mellanox NICs). Note that SQEs are allocated externally to the SQ descriptor ring (the same is not true for CQEs). SQEs roughly correspond to a single system call: they are tagged with an operation type, and filled in with the values that would traditionally be supplied as arguments to the appropriate function. Userspace is provided references to SQEs on the SQE ring, which it fills in and submits. Submission operates up through a specified SQE, and thus all SQEs before it in the ring must also be ready to go (this is likely the main reason why the SQ holds descriptors to an external ring of SQEs: you can acquire SQEs, and then submit them out of order, but see <tt>IORING_SETUP_NO_SQARRAY</tt>). The kernel places results in the CQE ring. These rings are shared between kernel- and userspace. The rings must be distinct unless the kernel specifies the <tt>IORING_FEAT_SINGLE_MMAP</tt> feature (see below).

uring does not generally make use of <tt>errno</tt>. Synchronous functions return the negative error code as their result. Completion queue entries have the negated error code placed in their <tt>res</tt> fields.

Line 57:

|-

| <tt>IORING_ENTER_REGISTERED_RING</tt> || <tt>ring_fd</tt> is an offset into the registered ring pool rather than a normal file descriptor.

|-

| <tt>IORING_ENTER_ABS_TIMER</tt> || (Since Linux 6.12) The timeout argument in the <tt>io_uring_getevents_arg</tt> is an absolute time, using the registered clock.

|-

|}

Line 219:

Line 221:

|-

| IORING_SETUP_REGISTERED_FD_ONLY || 6.6 || Registers the ring fd for use with <tt>IORING_REGISTER_USE_REGISTERED_RING</tt>. Returns a registered fd index.

|-

| IORING_SETUP_NO_SQARRAY || 6.6 || Do not use an indirection array for the submission queue, instead submitting them strictly in order.

|-

|}

Line 260:

Line 264:

|-

| <tt>IORING_FEAT_REG_REG_RING</tt> || 6.3 || Ring descriptors can be registered with <tt>IORING_REGISTER_USE_REGISTERED_RING</tt>.

|-

| <tt>IORING_FEAT_RECVSEND_BUNDLE</tt> || 6.10 || <tt>send</tt> and <tt>recv</tt> operations can be bundled.

|-

| <tt>IORING_FEAT_MIN_TIMEOUT</tt> || 6.12 || A minimum batch wait timeout is supported.

|-

|}

Line 359:

Line 367:

|-

| <tt>IORING_UNREGISTER_PERSONALITY</tt> || 5.6 ||

|-

|}

====NAPI settings====

{| class="wikitable" border="1"

! Flag !! Kernel !! Description

|-

| <tt>IORING_REGISTER_NAPI</tt> || ||

|-

| <tt>IORING_UNREGISTER_PERSONALITY</tt> || ||

|-

|}

Line 410:

Line 427:

|-

| <tt>IORING_RECV_MULTISHOT</tt> ||

|-

| <tt>IORING_SEND_MULTISHOT</tt> || Introduced in 6.8.

|-

| <tt>IORING_ACCEPT_MULTISHOT</tt> ||

Line 556:

Line 575:

// IORING_OP_CONNECT (5.5)

void io_uring_prep_connect(struct io_uring_sqe *sqe, int sockfd, const struct sockaddr *addr, socklen_t addrlen);

</syntaxhighlight>

====Operations on file descriptors====

// IORING_OP_URING_CMD

void io_uring_prep_cmd_sock(struct io_uring_sqe *sqe, int cmd_op, int fd, int level,

int optname, void *optval, int optlen);

// IORING_OP_GETXATTR

void io_uring_prep_getxattr(struct io_uring_sqe *sqe, const char *name, char *value,

const char *path, unsigned int len);

// IORING_OP_SETXATTR

void io_uring_prep_setxattr(struct io_uring_sqe *sqe, const char *name, const char *value,

const char *path, int flags, unsigned int len);

// IORING_OP_FGETXATTR

void io_uring_prep_fgetxattr(struct io_uring_sqe *sqe, int fd, const char *name,

char *value, unsigned int len);

// IORING_OP_FSETXATTR

void io_uring_prep_fsetxattr(struct io_uring_sqe *sqe, int fd, const char *name,

const char *value, int flags, unsigned int len);

</syntaxhighlight>

Line 596:

Line 634:

All transring messages are prepared using the target uring's file descriptor, and thus aren't particularly compatible with the higher-level liburing API (of course, you can just yank <tt>ring_fd</tt> out of <tt>struct iouring</tt>). That's unfortunate, because this functionality is very useful in a multithreaded environment. Note that the ring to which the SQE is submitted can itself be the target of the message!

// IORING_OP_MSG_RING

void io_uring_prep_msg_ring(struct io_uring_sqe *sqe, int fd, unsigned int len, __u64 data, unsigned int flags);

void io_uring_prep_msg_ring_cqe_flags(struct io_uring_sqe *sqe, int fd, unsigned int len, __u64 data, unsigned int flags, unsigned int cqe_flags);

void io_uring_prep_msg_ring_fd(struct io_uring_sqe *sqe, int fd, int source_fd, int target_fd, __u64 data, unsigned int flags);

void io_uring_prep_msg_ring_fd_alloc(struct io_uring_sqe *sqe, int fd, int source_fd, __u64 data, unsigned int flags);

</syntaxhighlight>

====Futexes====

// IORING_OP_FUTEX_WAKE

void io_uring_prep_futex_wake(struct io_uring_sqe *sqe, uint32_t *futex, uint64_t val,

uint64_t mask, uint32_t futex_flags, unsigned int flags);

// IORING_OP_FUTEX_WAIT

void io_uring_prep_futex_wait(struct io_uring_sqe *sqe, uint32_t *futex, uint64_t val,

uint64_t mask, uint32_t futex_flags, unsigned int flags);

// IORING_OP_FUTEX_WAITV

void io_uring_prep_futex_waitv(struct io_uring_sqe *sqe, struct futex_waitv *futex,

uint32_t nr_futex, unsigned int flags);

</syntaxhighlight>

Line 688:

Line 740:

|-

| <tt>IORING_CQE_F_NOTIF</tt> || Notification CQE for zero-copy sends

|-

| <tt>IORING_CQE_F_BUF_MORE</tt> || There will be more CQEs to the specified registered buffer

|-

|}

Line 776:

Line 830:

Why is there no <tt>vmsplice(2)</tt> action? How about <tt>fork(2)</tt>/<tt>pthread_create(3)</tt> (or more generally <tt>clone(2)</tt>)? We could have coroutines with I/O.

I understand that making huge pages as unpleasant as possible to use is a central tenet (perhaps <i>the</i> central tenet) underpinning the entire Linux mission, but it would be great if hugepage-backed urings were only a flag/field away (<b>UPDATE 2023-06-30:</b> IORING_SETUP_NO_MMAP has been [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=03d89a2de25b added for 6.5], huzzah!).

It's kind of annoying that chains can't extend over submissions. If I've got a lot of data I need delivered in order, it seems I'm limited to a single chain in-flight at a time, or else I risk out-of-order delivery due to one chain aborting, followed by items from a subsequent chain succeeding. <tt>recv_multishot</tt> can help with this.

Line 803:

Line 855:

** "[https://lwn.net/Articles/863071/ Discriptorless files for io_uring]" 2021-07-19

** "[https://lwn.net/Articles/879724/ Zero-copy network transmission with io_uring]" 2021-12-30

** "[https://lwn.net/Articles/1002371/ Process creation in io_uring]" 2024-12-20

[[CATEGORY: Networking]]

Io uring: Difference between revisions

Dank (talk | contribs)

@@ Line 11: / Line 11: @@
 ==Rings==
-Central to every uring are two ringbuffers holding CQEs (Completion Queue Entries) and SQE (Submission Queue Entries) descriptors (as best I can tell, this terminology was borrowed from the [https://nvmexpress.org/specifications/ NVMe specification]; I've also seen it with Mellanox NICs). Note that SQEs are allocated externally to the SQ descriptor ring (the same is not true for CQEs). SQEs roughly correspond to a single system call: they are tagged with an operation type, and filled in with the values that would traditionally be supplied as arguments to the appropriate function. Userspace is provided references to SQEs on the SQE ring, which it fills in and submits. Submission operates up through a specified SQE, and thus all SQEs before it in the ring must also be ready to go (this is likely the main reason why the SQ holds descriptors to an external ring of SQEs: you can acquire SQEs, and then submit them out of order). The kernel places results in the CQE ring. These rings are shared between kernel- and userspace. The rings must be distinct unless the kernel specifies the <tt>IORING_FEAT_SINGLE_MMAP</tt> feature (see below).
+Central to every uring are two ringbuffers holding CQEs (Completion Queue Entries) and SQE (Submission Queue Entries) descriptors (as best I can tell, this terminology was borrowed from the [https://nvmexpress.org/specifications/ NVMe specification]; I've also seen it with Mellanox NICs). Note that SQEs are allocated externally to the SQ descriptor ring (the same is not true for CQEs). SQEs roughly correspond to a single system call: they are tagged with an operation type, and filled in with the values that would traditionally be supplied as arguments to the appropriate function. Userspace is provided references to SQEs on the SQE ring, which it fills in and submits. Submission operates up through a specified SQE, and thus all SQEs before it in the ring must also be ready to go (this is likely the main reason why the SQ holds descriptors to an external ring of SQEs: you can acquire SQEs, and then submit them out of order, but see <tt>IORING_SETUP_NO_SQARRAY</tt>). The kernel places results in the CQE ring. These rings are shared between kernel- and userspace. The rings must be distinct unless the kernel specifies the <tt>IORING_FEAT_SINGLE_MMAP</tt> feature (see below).
 uring does not generally make use of <tt>errno</tt>. Synchronous functions return the negative error code as their result. Completion queue entries have the negated error code placed in their <tt>res</tt> fields.
@@ Line 57: / Line 57: @@
 |-
 | <tt>IORING_ENTER_REGISTERED_RING</tt> || <tt>ring_fd</tt> is an offset into the registered ring pool rather than a normal file descriptor.
+|-
+| <tt>IORING_ENTER_ABS_TIMER</tt> || (Since Linux 6.12) The timeout argument in the <tt>io_uring_getevents_arg</tt> is an absolute time, using the registered clock.
 |-
 |}
@@ Line 219: / Line 221: @@
 |-
 | IORING_SETUP_REGISTERED_FD_ONLY || 6.6 || Registers the ring fd for use with <tt>IORING_REGISTER_USE_REGISTERED_RING</tt>. Returns a registered fd index.
+|-
+| IORING_SETUP_NO_SQARRAY || 6.6 || Do not use an indirection array for the submission queue, instead submitting them strictly in order.
 |-
 |}
@@ Line 260: / Line 264: @@
 |-
 | <tt>IORING_FEAT_REG_REG_RING</tt> || 6.3 || Ring descriptors can be registered with <tt>IORING_REGISTER_USE_REGISTERED_RING</tt>.
+|-
+| <tt>IORING_FEAT_RECVSEND_BUNDLE</tt> || 6.10 || <tt>send</tt> and <tt>recv</tt> operations can be bundled.
+|-
+| <tt>IORING_FEAT_MIN_TIMEOUT</tt> || 6.12 || A minimum batch wait timeout is supported.
 |-
 |}
@@ Line 359: / Line 367: @@
 |-
 | <tt>IORING_UNREGISTER_PERSONALITY</tt> || 5.6 ||
+|-
+|}
+====NAPI settings====
+{| class="wikitable" border="1"
+! Flag !! Kernel !! Description
+|-
+| <tt>IORING_REGISTER_NAPI</tt> ||  ||
+|-
+| <tt>IORING_UNREGISTER_PERSONALITY</tt> ||  ||
 |-
 |}
@@ Line 410: / Line 427: @@
 |-
 | <tt>IORING_RECV_MULTISHOT</tt> ||
+|-
+| <tt>IORING_SEND_MULTISHOT</tt> || Introduced in 6.8.
 |-
 | <tt>IORING_ACCEPT_MULTISHOT</tt> ||
@@ Line 556: / Line 575: @@
 // IORING_OP_CONNECT (5.5)
 void io_uring_prep_connect(struct io_uring_sqe *sqe, int sockfd, const struct sockaddr *addr, socklen_t addrlen);
+</syntaxhighlight>
+====Operations on file descriptors====
+<syntaxhighlight lang="c">
+// IORING_OP_URING_CMD
+void io_uring_prep_cmd_sock(struct io_uring_sqe *sqe, int cmd_op, int fd, int level,
+                            int optname, void *optval, int optlen);
+// IORING_OP_GETXATTR
+void io_uring_prep_getxattr(struct io_uring_sqe *sqe, const char *name, char *value,
+                            const char *path, unsigned int len);
+// IORING_OP_SETXATTR
+void io_uring_prep_setxattr(struct io_uring_sqe *sqe, const char *name, const char *value,
+                            const char *path, int flags, unsigned int len);
+// IORING_OP_FGETXATTR
+void io_uring_prep_fgetxattr(struct io_uring_sqe *sqe, int fd, const char *name,
+                             char *value, unsigned int len);
+// IORING_OP_FSETXATTR
+void io_uring_prep_fsetxattr(struct io_uring_sqe *sqe, int fd, const char *name,
+                             const char *value, int flags, unsigned int len);
 </syntaxhighlight>
@@ Line 596: / Line 634: @@
 All transring messages are prepared using the target uring's file descriptor, and thus aren't particularly compatible with the higher-level liburing API (of course, you can just yank <tt>ring_fd</tt> out of <tt>struct iouring</tt>). That's unfortunate, because this functionality is very useful in a multithreaded environment. Note that the ring to which the SQE is submitted can itself be the target of the message!
 <syntaxhighlight lang="c">
+// IORING_OP_MSG_RING
 void io_uring_prep_msg_ring(struct io_uring_sqe *sqe, int fd, unsigned int len, __u64 data, unsigned int flags);
 void io_uring_prep_msg_ring_cqe_flags(struct io_uring_sqe *sqe, int fd, unsigned int len, __u64 data, unsigned int flags, unsigned int cqe_flags);
 void io_uring_prep_msg_ring_fd(struct io_uring_sqe *sqe, int fd, int source_fd, int target_fd, __u64 data, unsigned int flags);
 void io_uring_prep_msg_ring_fd_alloc(struct io_uring_sqe *sqe, int fd, int source_fd, __u64 data, unsigned int flags);
+</syntaxhighlight>
+====Futexes====
+<syntaxhighlight lang="c">
+// IORING_OP_FUTEX_WAKE
+void io_uring_prep_futex_wake(struct io_uring_sqe *sqe, uint32_t *futex, uint64_t val,
+                              uint64_t mask, uint32_t futex_flags, unsigned int flags);
+// IORING_OP_FUTEX_WAIT
+void io_uring_prep_futex_wait(struct io_uring_sqe *sqe, uint32_t *futex, uint64_t val,
+                              uint64_t mask, uint32_t futex_flags, unsigned int flags);
+// IORING_OP_FUTEX_WAITV
+void io_uring_prep_futex_waitv(struct io_uring_sqe *sqe, struct futex_waitv *futex,
+                               uint32_t nr_futex, unsigned int flags);
 </syntaxhighlight>
@@ Line 688: / Line 740: @@
 |-
 | <tt>IORING_CQE_F_NOTIF</tt> || Notification CQE for zero-copy sends
+|-
+| <tt>IORING_CQE_F_BUF_MORE</tt> || There will be more CQEs to the specified registered buffer
 |-
 |}
@@ Line 776: / Line 830: @@
 Why is there no <tt>vmsplice(2)</tt> action? How about <tt>fork(2)</tt>/<tt>pthread_create(3)</tt> (or more generally <tt>clone(2)</tt>)? We could have coroutines with I/O.
-I understand that making huge pages as unpleasant as possible to use is a central tenet (perhaps <i>the</i> central tenet) underpinning the entire Linux mission, but it would be great if hugepage-backed urings were only a flag/field away (<b>UPDATE 2023-06-30:</b> IORING_SETUP_NO_MMAP has been [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=03d89a2de25b added for 6.5], huzzah!).
 It's kind of annoying that chains can't extend over submissions. If I've got a lot of data I need delivered in order, it seems I'm limited to a single chain in-flight at a time, or else I risk out-of-order delivery due to one chain aborting, followed by items from a subsequent chain succeeding. <tt>recv_multishot</tt> can help with this.
@@ Line 803: / Line 855: @@
 ** "[https://lwn.net/Articles/863071/ Discriptorless files for io_uring]" 2021-07-19
 ** "[https://lwn.net/Articles/879724/ Zero-copy network transmission with io_uring]" 2021-12-30
+** "[https://lwn.net/Articles/1002371/ Process creation in io_uring]" 2024-12-20
 [[CATEGORY: Networking]]