Io uring

io_uring, introduced in 2019 by Jens Axboe, is a system for providing the kernel with a schedule of system calls, and receiving the results as they're generated. It combines asynchronous I/O, system call polybatching, and flexible buffer management, and is IMHO the most substantial development in the Linux I/O model since Berkeley sockets:

• Asynchronous I/O without the large copy overheads and restrictions of POSIX AIO (no more O_DIRECT, etc.)
• System call batching across distinct system calls (not just readv() and recvmmsg())
  • Whole sequences of distinct system calls can be strung together
• Provide a buffer pool, and they'll be used as needed

The core system calls of io_uring are wrapped by the C API of liburing.

Rings

Central to every uring are two ringbuffers holding CQEs (Completion Queue Entries) and SQEs (Submission Queue Entries). SQEs roughly correspond to a single system call: they are tagged with an operation type, and filled in with the values that would traditionally be supplied as arguments to the appropriate function. Userspace is provided references to SQEs on the SQE ring, filled in, and submitted. Submission operates up through a specified SQE, and thus all SQEs before it in the ring must also be ready to go. The kernel places results in the CQE ring. These rings are shared between kernel- and userspace. The rings must be distinct unless the kernel specifies the IORING_FEAT_SINGLE_MMAP feature (see below).

It is possible for a single submission to result in multiple completions (e.g. io_uring_prep_multishot_accept(3)); this is known as multishot.

uring does not generally make use of errno. Synchronous functions return the negative error code as their result. Completion queue entries have the negated error code placed in their res fields.

Setup

The io_uring_setup(2) system call returns a file descriptor, and accepts two parameters, u32 entries and struct io_uring_params *p:

int io_uring_setup(u32 entries, struct io_uring_params *p);
struct io_uring_params {                                                                                                            
  __u32 sq_entries;                                                                                                                 
  __u32 cq_entries;                                                                                                                 
  __u32 flags;                                                                                                                      
  __u32 sq_thread_cpu;                                                                                                              
  __u32 sq_thread_idle;                                                                                                             
  __u32 features;                                                                                                                   
  __u32 wq_fd;                                                                                                                      
  __u32 resv[3];                                                                                                                    
  struct io_sqring_offsets sq_off;                                                                                                  
  struct io_cqring_offsets cq_off;                                                                                                  
};

It is wrapped by liburing's io_uring_queue_init(3) and io_uring_queue_init_params(3). When using these wrappers, io_uring_queue_exit(3) should be used to clean up.

Kernel features

Various functionality was added to the kernel following the initial release of uring, and thus not necessarily available to all kernels supporting the basic system calls. The __u32 features field of the io_uring_params parameter to io_uring_setup(2) is filled in with feature flags by the kernel.

Links

• Efficient IO with io_uring
• Lord of the io_uring