Io uring

(25 intermediate revisions by the same user not shown)

Line 11:

==Rings==

Central to every uring are two ringbuffers holding CQEs (Completion Queue Entries) and SQE (Submission Queue Entries) descriptors (as best I can tell, this terminology was borrowed from the [https://nvmexpress.org/specifications/ NVMe specification]; I've also seen it with Mellanox NICs). Note that SQEs are allocated externally to the SQ descriptor ring (the same is not true for CQEs). SQEs roughly correspond to a single system call: they are tagged with an operation type, and filled in with the values that would traditionally be supplied as arguments to the appropriate function. Userspace is provided references to SQEs on the SQE ring, which it fills in and submits. Submission operates up through a specified SQE, and thus all SQEs before it in the ring must also be ready to go (this is likely the main reason why the SQ holds descriptors to an external ring of SQEs: you can acquire SQEs, and then submit them out of order). The kernel places results in the CQE ring. These rings are shared between kernel- and userspace. The rings must be distinct unless the kernel specifies the <tt>IORING_FEAT_SINGLE_MMAP</tt> feature (see below).

Central to every uring are two ringbuffers holding CQEs (Completion Queue Entries) and SQE (Submission Queue Entries) descriptors (as best I can tell, this terminology was borrowed from the [https://nvmexpress.org/specifications/ NVMe specification]; I've also seen it with Mellanox NICs). Note that SQEs are allocated externally to the SQ descriptor ring (the same is not true for CQEs). SQEs roughly correspond to a single system call: they are tagged with an operation type, and filled in with the values that would traditionally be supplied as arguments to the appropriate function. Userspace is provided references to SQEs on the SQE ring, which it fills in and submits. Submission operates up through a specified SQE, and thus all SQEs before it in the ring must also be ready to go (this is likely the main reason why the SQ holds descriptors to an external ring of SQEs: you can acquire SQEs, and then submit them out of order, but see <tt>IORING_SETUP_NO_SQARRAY</tt>). The kernel places results in the CQE ring. These rings are shared between kernel- and userspace. The rings must be distinct unless the kernel specifies the <tt>IORING_FEAT_SINGLE_MMAP</tt> feature (see below).

uring does not generally make use of <tt>errno</tt>. Synchronous functions return the negative error code as their result. Completion queue entries have the negated error code placed in their <tt>res</tt> fields.

Line 57:

|-

| <tt>IORING_ENTER_REGISTERED_RING</tt> || <tt>ring_fd</tt> is an offset into the registered ring pool rather than a normal file descriptor.

|-

| <tt>IORING_ENTER_ABS_TIMER</tt> || (Since Linux 6.12) The timeout argument in the <tt>io_uring_getevents_arg</tt> is an absolute time, using the registered clock.

|-

|}

Line 81:

Line 83:

<tt>resv</tt> must be zeroed out. In the absence of flags, the uring uses interrupt-driven I/O. Calling <tt>close(2)</tt> on the returned descriptor frees all resources associated with the uring.

<tt>io_uring_setup(2)</tt> is wrapped by liburing's <tt>io_uring_queue_init(3)</tt> and <tt>io_uring_queue_init_params(3)</tt>. When using these wrappers, <tt>io_uring_queue_exit(3)</tt> should be used to clean up. ~~These wrappers operate on a <tt>struct io_uring</tt>:~~

<tt>io_uring_setup(2)</tt> is wrapped by liburing's <tt>io_uring_queue_init(3)</tt> and <tt>io_uring_queue_init_params(3)</tt>. When using these wrappers, <tt>io_uring_queue_exit(3)</tt> should be used to clean up.

struct io_uring {

int io_uring_queue_init(unsigned entries, struct io_uring *ring, unsigned flags);

~~struct io_uring_sq sq;~~

int io_uring_queue_init_params(unsigned entries, struct io_uring *ring, struct io_uring_params *params);

~~struct io_uring_cq cq;~~

void io_uring_queue_exit(struct io_uring *ring);

unsigned flags;

int ~~ring_fd;~~

unsigned ~~features~~;

~~int enter_ring_fd;~~

~~__u8 int_flags;~~

~~__u8 pad[3];~~

~~unsigned pad2;~~

};

</syntaxhighlight>

~~<tt>io_uring_queue_init(3)</tt> takes an <tt>unsigned flags</tt> argument, which is passed as the <tt>flags</tt> field of <tt>io_uring_params</tt>. <tt>io_uring_queue_init_params(3)</tt> takes~~ a <tt>struct io_uring_params*</tt> ~~argument, which is passed through directly to <tt>io_uring_setup(2)</tt>. It's best to avoid mixing the low-level API and that provided by liburing.~~

These wrappers operate on a <tt>struct io_uring</tt>:

~~===Ring structure===~~

The details of ring structure are typically only relevant when using the low-level API. Two offset structs are used to prepare and control the three (or two, see <tt>IORING_FEAT_SINGLE_MMAP</tt>) backing memory maps.

struct io_uring {

struct io_uring_sq sq;

struct io_uring_cq cq;

unsigned flags;

int ring_fd;

unsigned features;

int enter_ring_fd;

__u8 int_flags;

__u8 pad[3];

unsigned pad2;

};

struct io_uring_sq {

unsigned *khead;

unsigned *ktail;

// Deprecated: use `ring_mask~~` instead of `*kring_mask~~`

unsigned *kring_mask; // Deprecated: use `ring_mask`

unsigned *~~kring_mask~~;

unsigned *kring_entries; // Deprecated: use `ring_entries`

// Deprecated: use `ring_entries` ~~instead of `*kring_entries`~~

~~unsigned *kring_entries;~~

unsigned *kflags;

unsigned *kdropped;

unsigned *array;

struct io_uring_sqe *sqes;

unsigned sqe_head;

unsigned sqe_tail;

size_t ring_sz;

void *ring_ptr;

unsigned ring_mask;

unsigned ring_entries;

unsigned pad[2];

};

Line 128:

Line 127:

unsigned *khead;

unsigned *ktail;

// Deprecated: use `ring_mask~~` instead of `*kring_mask~~`

unsigned *kring_mask; // Deprecated: use `ring_mask`

unsigned *~~kring_mask~~;

unsigned *kring_entries; // Deprecated: use `ring_entries`

// Deprecated: use `ring_entries` ~~instead of `*kring_entries`~~

~~unsigned *kring_entries;~~

unsigned *kflags;

unsigned *koverflow;

struct io_uring_cqe *cqes;

size_t ring_sz;

void *ring_ptr;

unsigned ring_mask;

unsigned ring_entries;

unsigned pad[2];

};

</syntaxhighlight>

<tt>io_uring_queue_init(3)</tt> takes an <tt>unsigned flags</tt> argument, which is passed as the <tt>flags</tt> field of <tt>io_uring_params</tt>. <tt>io_uring_queue_init_params(3)</tt> takes a <tt>struct io_uring_params*</tt> argument, which is passed through directly to <tt>io_uring_setup(2)</tt>. It's best to avoid mixing the low-level API and that provided by liburing.

===Ring structure===

The details of ring structure are typically only relevant when using the low-level API. Two offset structs are used to prepare and control the three (or two, see <tt>IORING_FEAT_SINGLE_MMAP</tt>) backing memory maps.

struct io_sqring_offsets {

__u32 head;

Line 220:

Line 219:

|-

| IORING_SETUP_NO_MMAP || 6.5 || The caller will provide two memory regions (the queue rings in <tt>cq_off.user_data</tt>, the SQEs in <tt>sq_off.user_data</tt>) rather than having them allocated by the kernel (they may both come from a single map). This allows for the use of huge pages.

|-

| IORING_SETUP_REGISTERED_FD_ONLY || 6.6 || Registers the ring fd for use with <tt>IORING_REGISTER_USE_REGISTERED_RING</tt>. Returns a registered fd index.

|-

| IORING_SETUP_NO_SQARRAY || 6.6 || Do not use an indirection array for the submission queue, instead submitting them strictly in order.

|-

|}

Line 261:

Line 264:

|-

| <tt>IORING_FEAT_REG_REG_RING</tt> || 6.3 || Ring descriptors can be registered with <tt>IORING_REGISTER_USE_REGISTERED_RING</tt>.

|-

| <tt>IORING_FEAT_RECVSEND_BUNDLE</tt> || 6.10 || <tt>send</tt> and <tt>recv</tt> operations can be bundled.

|-

| <tt>IORING_FEAT_MIN_TIMEOUT</tt> || 6.12 || A minimum batch wait timeout is supported.

|-

|}

Line 360:

Line 367:

|-

| <tt>IORING_UNREGISTER_PERSONALITY</tt> || 5.6 ||

|-

|}

====NAPI settings====

{| class="wikitable" border="1"

! Flag !! Kernel !! Description

|-

| <tt>IORING_REGISTER_NAPI</tt> || ||

|-

| <tt>IORING_UNREGISTER_PERSONALITY</tt> || ||

|-

|}

Line 411:

Line 427:

|-

| <tt>IORING_RECV_MULTISHOT</tt> ||

|-

| <tt>IORING_SEND_MULTISHOT</tt> || Introduced in 6.8.

|-

| <tt>IORING_ACCEPT_MULTISHOT</tt> ||

Line 557:

Line 575:

// IORING_OP_CONNECT (5.5)

void io_uring_prep_connect(struct io_uring_sqe *sqe, int sockfd, const struct sockaddr *addr, socklen_t addrlen);

</syntaxhighlight>

====Operations on file descriptors====

// IORING_OP_URING_CMD

void io_uring_prep_cmd_sock(struct io_uring_sqe *sqe, int cmd_op, int fd, int level,

int optname, void *optval, int optlen);

// IORING_OP_GETXATTR

void io_uring_prep_getxattr(struct io_uring_sqe *sqe, const char *name, char *value,

const char *path, unsigned int len);

// IORING_OP_SETXATTR

void io_uring_prep_setxattr(struct io_uring_sqe *sqe, const char *name, const char *value,

const char *path, int flags, unsigned int len);

// IORING_OP_FGETXATTR

void io_uring_prep_fgetxattr(struct io_uring_sqe *sqe, int fd, const char *name,

char *value, unsigned int len);

// IORING_OP_FSETXATTR

void io_uring_prep_fsetxattr(struct io_uring_sqe *sqe, int fd, const char *name,

const char *value, int flags, unsigned int len);

</syntaxhighlight>

Line 597:

Line 634:

All transring messages are prepared using the target uring's file descriptor, and thus aren't particularly compatible with the higher-level liburing API (of course, you can just yank <tt>ring_fd</tt> out of <tt>struct iouring</tt>). That's unfortunate, because this functionality is very useful in a multithreaded environment. Note that the ring to which the SQE is submitted can itself be the target of the message!

// IORING_OP_MSG_RING

void io_uring_prep_msg_ring(struct io_uring_sqe *sqe, int fd, unsigned int len, __u64 data, unsigned int flags);

void io_uring_prep_msg_ring_cqe_flags(struct io_uring_sqe *sqe, int fd, unsigned int len, __u64 data, unsigned int flags, unsigned int cqe_flags);

void io_uring_prep_msg_ring_fd(struct io_uring_sqe *sqe, int fd, int source_fd, int target_fd, __u64 data, unsigned int flags);

void io_uring_prep_msg_ring_fd_alloc(struct io_uring_sqe *sqe, int fd, int source_fd, __u64 data, unsigned int flags);

</syntaxhighlight>

====Futexes====

// IORING_OP_FUTEX_WAKE

void io_uring_prep_futex_wake(struct io_uring_sqe *sqe, uint32_t *futex, uint64_t val,

uint64_t mask, uint32_t futex_flags, unsigned int flags);

// IORING_OP_FUTEX_WAIT

void io_uring_prep_futex_wait(struct io_uring_sqe *sqe, uint32_t *futex, uint64_t val,

uint64_t mask, uint32_t futex_flags, unsigned int flags);

// IORING_OP_FUTEX_WAITV

void io_uring_prep_futex_waitv(struct io_uring_sqe *sqe, struct futex_waitv *futex,

uint32_t nr_futex, unsigned int flags);

</syntaxhighlight>

Line 689:

Line 740:

|-

| <tt>IORING_CQE_F_NOTIF</tt> || Notification CQE for zero-copy sends

|-

| <tt>IORING_CQE_F_BUF_MORE</tt> || There will be more CQEs to the specified registered buffer

|-

|}

Line 777:

Line 830:

Why is there no <tt>vmsplice(2)</tt> action? How about <tt>fork(2)</tt>/<tt>pthread_create(3)</tt> (or more generally <tt>clone(2)</tt>)? We could have coroutines with I/O.

It would be nice to have tight integration with condition variables or even mutex/futex (allow me to submit a request to get a lock, and when i get the CQE, i have that lock). Bonus points if the fast (uncontended) path never needs a system call (like mutexes built atop futexes today).

I understand that making huge pages as unpleasant as possible to use is a central tenet (perhaps the central tenet) underpinning the entire Linux mission, but it would be great if hugepage-backed urings were only a flag/field away (UPDATE 2023-06-30: IORING_SETUP_NO_MMAP has been [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=03d89a2de25b added for 6.5], huzzah!).

It's kind of annoying that chains can't extend over submissions. If I've got a lot of data I need delivered in order, it seems I'm limited to a single chain in-flight at a time, or else I risk out-of-order delivery due to one chain aborting, followed by items from a subsequent chain succeeding. <tt>recv_multishot</tt> can help with this.

Line 786:

Line 835:

It seems that the obvious next step would be providing small snippets of arbitrary computation to be run in kernelspace, linked with SQEs. Perhaps [[eBPF]] would be a good starting place.

It would be awesome if I could indicate in each SQE some other uring to which the CQE ought be delivered, basically combining inter-ring messages and SQE submission. This would be the ne plus ultra IMHO of multithreaded uring (and be very much like my [[Libtorque|masters thesis]], not that I claim any inspiration whatsoever for uring).

It would be awesome if I could indicate in each SQE some other uring to which the CQE ought be delivered, basically combining inter-ring messages and SQE submission. This would be the ne plus ultra IMHO of multithreaded uring (and be very much like my [[Libtorque|masters thesis]], not that I claim any inspiration/precedence whatsoever for uring).

==See also==

Line 806:

Line 855:

** "[https://lwn.net/Articles/863071/ Discriptorless files for io_uring]" 2021-07-19

** "[https://lwn.net/Articles/879724/ Zero-copy network transmission with io_uring]" 2021-12-30

** "[https://lwn.net/Articles/1002371/ Process creation in io_uring]" 2024-12-20

[[CATEGORY: Networking]]

Io uring: Difference between revisions

Dank (talk | contribs)

@@ Line 11: / Line 11: @@
 ==Rings==
-Central to every uring are two ringbuffers holding CQEs (Completion Queue Entries) and SQE (Submission Queue Entries) descriptors (as best I can tell, this terminology was borrowed from the [https://nvmexpress.org/specifications/ NVMe specification]; I've also seen it with Mellanox NICs). Note that SQEs are allocated externally to the SQ descriptor ring (the same is not true for CQEs). SQEs roughly correspond to a single system call: they are tagged with an operation type, and filled in with the values that would traditionally be supplied as arguments to the appropriate function. Userspace is provided references to SQEs on the SQE ring, which it fills in and submits. Submission operates up through a specified SQE, and thus all SQEs before it in the ring must also be ready to go (this is likely the main reason why the SQ holds descriptors to an external ring of SQEs: you can acquire SQEs, and then submit them out of order). The kernel places results in the CQE ring. These rings are shared between kernel- and userspace. The rings must be distinct unless the kernel specifies the <tt>IORING_FEAT_SINGLE_MMAP</tt> feature (see below).
+Central to every uring are two ringbuffers holding CQEs (Completion Queue Entries) and SQE (Submission Queue Entries) descriptors (as best I can tell, this terminology was borrowed from the [https://nvmexpress.org/specifications/ NVMe specification]; I've also seen it with Mellanox NICs). Note that SQEs are allocated externally to the SQ descriptor ring (the same is not true for CQEs). SQEs roughly correspond to a single system call: they are tagged with an operation type, and filled in with the values that would traditionally be supplied as arguments to the appropriate function. Userspace is provided references to SQEs on the SQE ring, which it fills in and submits. Submission operates up through a specified SQE, and thus all SQEs before it in the ring must also be ready to go (this is likely the main reason why the SQ holds descriptors to an external ring of SQEs: you can acquire SQEs, and then submit them out of order, but see <tt>IORING_SETUP_NO_SQARRAY</tt>). The kernel places results in the CQE ring. These rings are shared between kernel- and userspace. The rings must be distinct unless the kernel specifies the <tt>IORING_FEAT_SINGLE_MMAP</tt> feature (see below).
 uring does not generally make use of <tt>errno</tt>. Synchronous functions return the negative error code as their result. Completion queue entries have the negated error code placed in their <tt>res</tt> fields.
@@ Line 57: / Line 57: @@
 |-
 | <tt>IORING_ENTER_REGISTERED_RING</tt> || <tt>ring_fd</tt> is an offset into the registered ring pool rather than a normal file descriptor.
+|-
+| <tt>IORING_ENTER_ABS_TIMER</tt> || (Since Linux 6.12) The timeout argument in the <tt>io_uring_getevents_arg</tt> is an absolute time, using the registered clock.
 |-
 |}
@@ Line 81: / Line 83: @@
 <tt>resv</tt> must be zeroed out. In the absence of flags, the uring uses interrupt-driven I/O. Calling <tt>close(2)</tt> on the returned descriptor frees all resources associated with the uring.
-<tt>io_uring_setup(2)</tt> is wrapped by liburing's <tt>io_uring_queue_init(3)</tt> and <tt>io_uring_queue_init_params(3)</tt>. When using these wrappers, <tt>io_uring_queue_exit(3)</tt> should be used to clean up. These wrappers operate on a <tt>struct io_uring</tt>:
+<tt>io_uring_setup(2)</tt> is wrapped by liburing's <tt>io_uring_queue_init(3)</tt> and <tt>io_uring_queue_init_params(3)</tt>. When using these wrappers, <tt>io_uring_queue_exit(3)</tt> should be used to clean up.
 <syntaxhighlight lang="c">
-struct io_uring {
+int io_uring_queue_init(unsigned entries, struct io_uring *ring, unsigned flags);
-  struct io_uring_sq sq;
+int io_uring_queue_init_params(unsigned entries, struct io_uring *ring, struct io_uring_params *params);
-  struct io_uring_cq cq;
+void io_uring_queue_exit(struct io_uring *ring);
-  unsigned flags;
-  int ring_fd;
-  unsigned features;
-  int enter_ring_fd;
-  __u8 int_flags;
-  __u8 pad[3];
-  unsigned pad2;
-};
 </syntaxhighlight>
-<tt>io_uring_queue_init(3)</tt> takes an <tt>unsigned flags</tt> argument, which is passed as the <tt>flags</tt> field of <tt>io_uring_params</tt>. <tt>io_uring_queue_init_params(3)</tt> takes a <tt>struct io_uring_params*</tt> argument, which is passed through directly to <tt>io_uring_setup(2)</tt>. It's best to avoid mixing the low-level API and that provided by liburing.
+These wrappers operate on a <tt>struct io_uring</tt>:
-===Ring structure===
-The details of ring structure are typically only relevant when using the low-level API. Two offset structs are used to prepare and control the three (or two, see <tt>IORING_FEAT_SINGLE_MMAP</tt>) backing memory maps.
 <syntaxhighlight lang="c">
+struct io_uring {
+  struct io_uring_sq sq;
+  struct io_uring_cq cq;
+  unsigned flags;
+  int ring_fd;
+  unsigned features;
+  int enter_ring_fd;
+  __u8 int_flags;
+  __u8 pad[3];
+  unsigned pad2;
+};
 struct io_uring_sq {
    unsigned *khead;
    unsigned *ktail;
-   // Deprecated: use `ring_mask` instead of `*kring_mask`
+   unsigned *kring_mask;     // Deprecated: use `ring_mask`
-   unsigned *kring_mask;
+   unsigned *kring_entries;  // Deprecated: use `ring_entries`
-  // Deprecated: use `ring_entries` instead of `*kring_entries`
-  unsigned *kring_entries;
    unsigned *kflags;
    unsigned *kdropped;
    unsigned *array;
    struct io_uring_sqe *sqes;
    unsigned sqe_head;
    unsigned sqe_tail;
    size_t ring_sz;
    void *ring_ptr;
    unsigned ring_mask;
    unsigned ring_entries;
    unsigned pad[2];
 };
@@ Line 128: / Line 127: @@
    unsigned *khead;
    unsigned *ktail;
-   // Deprecated: use `ring_mask` instead of `*kring_mask`
+   unsigned *kring_mask;     // Deprecated: use `ring_mask`
-   unsigned *kring_mask;
+   unsigned *kring_entries;  // Deprecated: use `ring_entries`
-  // Deprecated: use `ring_entries` instead of `*kring_entries`
-  unsigned *kring_entries;
    unsigned *kflags;
    unsigned *koverflow;
    struct io_uring_cqe *cqes;
    size_t ring_sz;
    void *ring_ptr;
    unsigned ring_mask;
    unsigned ring_entries;
    unsigned pad[2];
 };
+</syntaxhighlight>
+<tt>io_uring_queue_init(3)</tt> takes an <tt>unsigned flags</tt> argument, which is passed as the <tt>flags</tt> field of <tt>io_uring_params</tt>. <tt>io_uring_queue_init_params(3)</tt> takes a <tt>struct io_uring_params*</tt> argument, which is passed through directly to <tt>io_uring_setup(2)</tt>. It's best to avoid mixing the low-level API and that provided by liburing.
+===Ring structure===
+The details of ring structure are typically only relevant when using the low-level API. Two offset structs are used to prepare and control the three (or two, see <tt>IORING_FEAT_SINGLE_MMAP</tt>) backing memory maps.
+<syntaxhighlight lang="c">
 struct io_sqring_offsets {
      __u32 head;
@@ Line 220: / Line 219: @@
 |-
 | IORING_SETUP_NO_MMAP || 6.5 || The caller will provide two memory regions (the queue rings in <tt>cq_off.user_data</tt>, the SQEs in <tt>sq_off.user_data</tt>) rather than having them allocated by the kernel (they may both come from a single map). This allows for the use of huge pages.
+|-
+| IORING_SETUP_REGISTERED_FD_ONLY || 6.6 || Registers the ring fd for use with <tt>IORING_REGISTER_USE_REGISTERED_RING</tt>. Returns a registered fd index.
+|-
+| IORING_SETUP_NO_SQARRAY || 6.6 || Do not use an indirection array for the submission queue, instead submitting them strictly in order.
 |-
 |}
@@ Line 261: / Line 264: @@
 |-
 | <tt>IORING_FEAT_REG_REG_RING</tt> || 6.3 || Ring descriptors can be registered with <tt>IORING_REGISTER_USE_REGISTERED_RING</tt>.
+|-
+| <tt>IORING_FEAT_RECVSEND_BUNDLE</tt> || 6.10 || <tt>send</tt> and <tt>recv</tt> operations can be bundled.
+|-
+| <tt>IORING_FEAT_MIN_TIMEOUT</tt> || 6.12 || A minimum batch wait timeout is supported.
 |-
 |}
@@ Line 360: / Line 367: @@
 |-
 | <tt>IORING_UNREGISTER_PERSONALITY</tt> || 5.6 ||
+|-
+|}
+====NAPI settings====
+{| class="wikitable" border="1"
+! Flag !! Kernel !! Description
+|-
+| <tt>IORING_REGISTER_NAPI</tt> ||  ||
+|-
+| <tt>IORING_UNREGISTER_PERSONALITY</tt> ||  ||
 |-
 |}
@@ Line 411: / Line 427: @@
 |-
 | <tt>IORING_RECV_MULTISHOT</tt> ||
+|-
+| <tt>IORING_SEND_MULTISHOT</tt> || Introduced in 6.8.
 |-
 | <tt>IORING_ACCEPT_MULTISHOT</tt> ||
@@ Line 557: / Line 575: @@
 // IORING_OP_CONNECT (5.5)
 void io_uring_prep_connect(struct io_uring_sqe *sqe, int sockfd, const struct sockaddr *addr, socklen_t addrlen);
+</syntaxhighlight>
+====Operations on file descriptors====
+<syntaxhighlight lang="c">
+// IORING_OP_URING_CMD
+void io_uring_prep_cmd_sock(struct io_uring_sqe *sqe, int cmd_op, int fd, int level,
+                            int optname, void *optval, int optlen);
+// IORING_OP_GETXATTR
+void io_uring_prep_getxattr(struct io_uring_sqe *sqe, const char *name, char *value,
+                            const char *path, unsigned int len);
+// IORING_OP_SETXATTR
+void io_uring_prep_setxattr(struct io_uring_sqe *sqe, const char *name, const char *value,
+                            const char *path, int flags, unsigned int len);
+// IORING_OP_FGETXATTR
+void io_uring_prep_fgetxattr(struct io_uring_sqe *sqe, int fd, const char *name,
+                             char *value, unsigned int len);
+// IORING_OP_FSETXATTR
+void io_uring_prep_fsetxattr(struct io_uring_sqe *sqe, int fd, const char *name,
+                             const char *value, int flags, unsigned int len);
 </syntaxhighlight>
@@ Line 597: / Line 634: @@
 All transring messages are prepared using the target uring's file descriptor, and thus aren't particularly compatible with the higher-level liburing API (of course, you can just yank <tt>ring_fd</tt> out of <tt>struct iouring</tt>). That's unfortunate, because this functionality is very useful in a multithreaded environment. Note that the ring to which the SQE is submitted can itself be the target of the message!
 <syntaxhighlight lang="c">
+// IORING_OP_MSG_RING
 void io_uring_prep_msg_ring(struct io_uring_sqe *sqe, int fd, unsigned int len, __u64 data, unsigned int flags);
 void io_uring_prep_msg_ring_cqe_flags(struct io_uring_sqe *sqe, int fd, unsigned int len, __u64 data, unsigned int flags, unsigned int cqe_flags);
 void io_uring_prep_msg_ring_fd(struct io_uring_sqe *sqe, int fd, int source_fd, int target_fd, __u64 data, unsigned int flags);
 void io_uring_prep_msg_ring_fd_alloc(struct io_uring_sqe *sqe, int fd, int source_fd, __u64 data, unsigned int flags);
+</syntaxhighlight>
+====Futexes====
+<syntaxhighlight lang="c">
+// IORING_OP_FUTEX_WAKE
+void io_uring_prep_futex_wake(struct io_uring_sqe *sqe, uint32_t *futex, uint64_t val,
+                              uint64_t mask, uint32_t futex_flags, unsigned int flags);
+// IORING_OP_FUTEX_WAIT
+void io_uring_prep_futex_wait(struct io_uring_sqe *sqe, uint32_t *futex, uint64_t val,
+                              uint64_t mask, uint32_t futex_flags, unsigned int flags);
+// IORING_OP_FUTEX_WAITV
+void io_uring_prep_futex_waitv(struct io_uring_sqe *sqe, struct futex_waitv *futex,
+                               uint32_t nr_futex, unsigned int flags);
 </syntaxhighlight>
@@ Line 689: / Line 740: @@
 |-
 | <tt>IORING_CQE_F_NOTIF</tt> || Notification CQE for zero-copy sends
+|-
+| <tt>IORING_CQE_F_BUF_MORE</tt> || There will be more CQEs to the specified registered buffer
 |-
 |}
@@ Line 777: / Line 830: @@
 Why is there no <tt>vmsplice(2)</tt> action? How about <tt>fork(2)</tt>/<tt>pthread_create(3)</tt> (or more generally <tt>clone(2)</tt>)? We could have coroutines with I/O.
-It would be nice to have tight integration with condition variables or even mutex/futex (allow me to submit a request to get a lock, and when i get the CQE, i have that lock). Bonus points if the fast (uncontended) path never needs a system call (like mutexes built atop futexes today).
-I understand that making huge pages as unpleasant as possible to use is a central tenet (perhaps <i>the</i> central tenet) underpinning the entire Linux mission, but it would be great if hugepage-backed urings were only a flag/field away (<b>UPDATE 2023-06-30:</b> IORING_SETUP_NO_MMAP has been [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=03d89a2de25b added for 6.5], huzzah!).
 It's kind of annoying that chains can't extend over submissions. If I've got a lot of data I need delivered in order, it seems I'm limited to a single chain in-flight at a time, or else I risk out-of-order delivery due to one chain aborting, followed by items from a subsequent chain succeeding. <tt>recv_multishot</tt> can help with this.
@@ Line 786: / Line 835: @@
 It seems that the obvious next step would be providing small snippets of arbitrary computation to be run in kernelspace, linked with SQEs. Perhaps [[eBPF]] would be a good starting place.
-It would be awesome if I could indicate in each SQE some other uring to which the CQE ought be delivered, basically combining inter-ring messages and SQE submission. This would be the <i>ne plus ultra</i> IMHO of multithreaded uring (and be very much like my [[Libtorque|masters thesis]], not that I claim any inspiration whatsoever for uring).
+It would be awesome if I could indicate in each SQE some other uring to which the CQE ought be delivered, basically combining inter-ring messages and SQE submission. This would be the <i>ne plus ultra</i> IMHO of multithreaded uring (and be very much like my [[Libtorque|masters thesis]], not that I claim any inspiration/precedence whatsoever for uring).
 ==See also==
@@ Line 806: / Line 855: @@
 ** "[https://lwn.net/Articles/863071/ Discriptorless files for io_uring]" 2021-07-19
 ** "[https://lwn.net/Articles/879724/ Zero-copy network transmission with io_uring]" 2021-12-30
+** "[https://lwn.net/Articles/1002371/ Process creation in io_uring]" 2024-12-20
 [[CATEGORY: Networking]]