Fast UNIX Servers

Everyone ought start with Dan Kegel's classic site, "The C10K Problem" (still updated from time to time). Jeff Darcy's "High-Performance Server Architecture" is much of the same. Everything here is advanced followup material to these excellent works, and of course the books of W. Richard Stevens.

Queueing Theory

• "Introduction to Queueing"
• Leonard Kleinrock's peerless Queueing Systems (Volume 1: Theory, Volume 2: Computer Applications)

Event Cores

• epoll on Linux, /dev/poll on Solaris, kqueue on FreeBSD
• liboop, libev and libevent
• Ulrich Drepper's "The Need for Aynchronous, ZeroCopy Network I/O"
  • If nothing else, Drepper's plans tend to become sudden and crushing realities in the glibc world

Edge and Level Triggering

• Historic interfaces like POSIX.1g/POSIX.1-2001's select(2) and POSIX.1-2001's poll(2) were level-triggered
• epoll (via EPOLLET) and kqueue (via EV_CLEAR) provide edge-triggered semantics
• fixme a thorough comparison of these is sorely needed

The Full Monty: A Theory of UNIX Servers

We must mix and match:

• Many event sources, of multiple types and possibly various triggering mechanisms (edge- vs level-triggered):
  • Socket descriptors, pipes
  • File descriptors referring to actual files (these usually have different blocking semantics)
  • Signals, perhaps being used for asynchronous I/O with descriptors (signalfd(2) on Linux unifies these with socket descriptors; kqueue supports EVFILT_SIGNAL events)
  • Timers (timerfd(2) on Linux unifies these with socket descriptors; kqueue supports EVFILT_TIMER events)
  • Condition variables and/or mutexes becoming available
  • Filesystem events (inotify(7) on Linux, EVFILT_VNODE with kqueue)
  • Networking events (netlink(7) (PF_NETLINK) sockets on Linux, EVFILT_NETDEV with kqueue)
• One or more event notifiers (epoll or kqueue fd)
• One or more event vectors, into which notifiers dump events
  • kqueue supports vectorized registration of event changes, extending the issue
• Threads -- one event notifier per? one shared event notifier with one event vector per? one shared event notifier feeding one shared event vector?

See Also

• "sendfile(): fairly sexy (nothing to do with ECN)" on lkml
• "mmap() sendfile()" on freebsd-hackers
• "sharing memory map between processes (same parent)" on comp.unix.programmer
• "some mmap observations compared to Linux 2.6/OpenBSD" on freebsd-hackers
• Stuart Cheshire's "Laws of Networkdynamics" and "It's the Latency, Stupid"
• "mremap help? or no support for FreeBSD?" on freebsd-hackers
• "Edge-triggered interfaces are too difficult?" on LWN, 2003-05-16
• "Edge- vs Level-Triggered Events on Pierre Phaneuf's livejournal (pphaneuf)
• "edge-triggered vs level-triggered epoll in kernel 2.6" on comp.unix.programmer, 2004-12-01