Pthreads

Dank's Sagelike Wisdom Concerning POSIX Threads

• The static initializers are unsung heroes. Learn them.
• If you are using asynchronous cancellation, you have a bug.
• If you are using pthread_attr_setstackaddr(), and you don't know why that's funny, you have a bug.
• Use of sched_yield() is almost certainly a bug, and furthermore it is womanly in nature.
  • This logic is trivially extended to pthread_yield().
• The Triforce of POSIX Thread stacks:
  • If you aren't allocating a stack fit to your thread, you will one day smash that stack.
  • If you're using your stack, you should probably heap-allocate or use TLS.
  • Recursion's all fun and games until your recurse on through to the other side.
  • Yes, this all applies to an unthreaded program (especially an embedded one), with a critical difference -- memory management is process-based, causing noisy failures in the unthreaded case. Meanwhile, a POSIX thread will happily push a frame down right over his buddy's DTV/TLS in NPTL (see NPTL, "Memory Allocation"). Have a good time debugging that, champ!
• Failure to check every return, especially pthread_mutex_destroy(), is cause for many many lashes of electric wire.
• Those who fail to designate an alternate signal stack will one day wish they had.
• Solaris 2.5's lack of timeslicing among compute-bound user threads can be addressed in two ways:
  • Setting the concurrency level to the number of compute-bound user threads + 1.
    • Surely this is not the Tao of Programming.
  • Spawning no more than one compute-bound user thread per CPU, and providing preemption.
    • Surely this is the Tao of Programming.
• Those who say, "I need never spawn thread one on Solaris 2.5" would trade their birthright for cold pottage.
• Before trying to solve an issue with POSIX thread scheduling parameters, see if you can't just drink some hydrochloric acid.
• No, you shouldn't be using the volatile keyword.

Condition variable clocks

By default, pthread_cond_timedwait() uses CLOCK_REALTIME, with all its attendant problems. You probably want it to use CLOCK_MONOTONIC. This can be effected on some implementations using pthread_condattr_setclock(). The new function pthread_cond_clockwait() has been proposed, but not yet accepted into the POSIX standard. Note that C++'s std::condition_variable also suffers from this braindamage.

Pthreads signal model

POSIX threads specify that all threads within a process share a PID, and that each has its own, distinct thread id. PIDs have the integer type pid_t, and only positive PIDs can define actual processes. TIDs have the opaque type pthread_t; they cannot be ordered, but can be compared for equality.

Other processes can direct signals only to a PID, not the thread IDs within another process (TIDs of a given process have no meaning outside that process). Within the process, signals can be directed to particular TIDs. Use of raise(s) in a threaded program is equivalent to pthread_kill(pthread_self(), s).

Signal dispositions (handlers) are per-process. Signal masks are per-thread, and inherited upon thread creation (sigprocmask() is undefined in threaded programs; pthread_sigmask() must be used). Alternate signal stacks descriptors are per-thread, and inherited upon thread creation.

Both the process as a whole and threads within that process have distinct sets of pending signals (signals which were blocked at the time of delivery). The former is composed of those process-directed signals which could not be dispatched to a single thread. The latter are made up of thread-directed signals which were being blocked by the specified thread.

A process-directed signal will be delivered to only one particular thread. For a single-threaded process, the demultiplexing is trivial and immediate; in this case, the process's and (single) thread's pending signals are the same. Otherwise, if threads are blocking on the signal in sigwait(), one of them is arbitrarily chosen. Otherwise, if threads are not blocking the signal, one of them is arbitrarily chosen. The signal otherwise remains pending across the process until it can be directed to a thread.

Synchronous signals, and those directed to a TID using pthread_kill() within the process, can be handled only by the specified thread. If the signal is masked by that thread, it will be marked pending.

POSIX cancellations are dispatched the same way as a thread-directed signal.

Signals consumed by sigwait() are not subsequently delivered, and thus any registered handlers will *not* be called.

kill() special case

When a thread directs a signal to its own process (using kill(p,s) where p == getpid()), and the calling thread is not blocking the signal, and the signal could not be delivered to any other thread, at least one pending signal (*not* necessarily s!) must be delivered to the calling thread before kill() returns.

Internally-used signals

If the threading library itself uses signals, they might not be safe for use by the application.

• NPTL: The first two realtime signals are used. SIGRTMIN is modified in system include files to accommodate this, so there will be no problems so long as signal numbers are never hardcoded.
• LinuxThreads on 2.2+: Same as NPTL, but three signal numbers are consumed.
• LinuxThreads before 2.0: SIGUSR1 and SIGUSR2 are used. The program must avoid use of SIGUSR1 and SIGUSR2 in either numeric or symbolic form.

LinuxThreads signals

LinuxThreads (or more accurately, clone() prior to the introduction of CLONE_PID) failed to guarantee shared PIDs among the threads of a process (each got their own). This severely compromised the POSIX signal model. No concept of process-directed signals exists in LinuxThreads, so external signals are directed to one thread. A "signal manager" thread launched by the library watched for threads terminated due to signal disposition, and attempted to kill the others. Killing the signal management thread was possible, and further broke signal delivery. Signals generated by the terminal, or delivered to the process group, were sent to all threads.

Compiling code with pthreads

First off, POSIX requires _REENTRANT be defined in the scope of all code run in a multithreaded manner. This is most typically achieved via -D_REENTRANT on the gcc command line. Some gcc versions / target architectures provide a -pthread option which sets -D_REENTRANT (as well as setting up pthread linking), as evidenced below:

Date: Sun, 8 Apr 2007 23:33:23 -0400
From: nick black <nick_black@securecomputing.com>
To: Sven Krasser <skrasser@securecomputing.com>
Cc: Nick Black <nblack@securecomputing.com>
Subject: Re: [repper-304] REENTRANT is required by POSIX for any code using pthreads

Sven Krasser rigorously showed:
> This is on Linux:
> skrasser@ctapd01:~$ cat test.c
> #ifdef _REENTRANT
> #error xxx
> #endif
> skrasser@ctapd01:~$ gcc -pthread test.c
> test.c:2:2: error: #error xxx

interesting. from the gcc pinfo page, "Index of Options":

* pthread <1>:                           SPARC Options.      (line  240)
* pthread <2>:                           RS/6000 and PowerPC Options.
                                                             (line  653)
* pthread:                               IA-64 Options.      (line  106)
* pthreads:                              SPARC Options.      (line  234)

how odd. i must investigate. in any case, yes, -pthread is definitely
defining _REEENTRANT here:

[diaconicon](0) $ touch t.c
[diaconicon](0) $ cpp -dM -pthread t.c > e 2>&1
[diaconicon](1) $ cpp -dM t.c > f 2>&1
[diaconicon](1) $ diff -ur f e
--- f   2007-04-08 23:31:12.000000000 -0400
+++ e   2007-04-08 23:31:05.000000000 -0400
@@ -26,6 +27,7 @@
 #define __DECIMAL_DIG__ 21
 #define __gnu_linux__ 1
 #define __LDBL_HAS_QUIET_NAN__ 1
+#define _REENTRANT 1
 #define __GNUC__ 4
 #define __DBL_MAX__ 1.7976931348623157e+308
 #define __DBL_HAS_INFINITY__ 1
[diaconicon](1) $

interesting. no matter what, you definitely need to be declaring
-D_THREAD_SAFE, though, as my previous example asserted.

off to dig through gcc source....

Nonetheless, this is a poorly-documented gcc-specific method; best to always provide -D_REENTRANT explicitly. -D_REENTRANT and other preprocessor directives which affect code selection should always precede -include options to gcc, ie:

gcc -D_REENTRANT -I/usr/local -include pthread.h ...

On FreeBSD's libc (and also possibly with regard to other third-part libraries), it's also necessary to define _THREAD_SAFE, as evidenced below (furthermore, you need _POSIX_PTHREAD_SEMANTICS and _P1003_1B_VISIBLE for reasons I determined long ago and have since forgotten, see the make snippet below --dank):

Newsgroups: sys.research.subversion.repper
From: Nick Black <nblack@securecomputing.com>
Subject: Re: [repper-304] REENTRANT is required by POSIX for any code using pthr
On 2007-04-08, Sven Krasser <skrasser@securecomputing.com> wrote:
> Hmm, doesn't -pthread do that?
> -pthread
>    Adds support for multithreading with the pthreads library.  This
>    option sets flags for both the preprocessor and linker.

where do you see this? from the gcc docs on freebsd 4.10-p24 (source:
/usr/src/contrib/gcc/gcc.1):

-pthread Link a user-threaded process against libc_r instead of libc.
         Objects linked into user-threaded processes  should  be
         compiled with -D_THREAD_SAFE.

[newdhcpbox](0) $ cat > g.c
#include <stdio.h>

feof(stderr)
[newdhcpbox](0) $ gcc -pthread -D_THREAD_SAFE -E g.c | grep stderr
extern FILE *__stdinp, *__stdoutp, *__stderrp;
feof((__stderrp) )
[newdhcpbox](0) $ gcc -pthread -E g.c | grep stderr
extern FILE *__stdinp, *__stdoutp, *__stderrp;
(((   (__stderrp)    )->_flags & 0x0020 ) != 0)
[newdhcpbox](0) $ grep -C3 feof /usr/include/stdio.h
__BEGIN_DECLS
void     clearerr __P((FILE *));
int      fclose __P((FILE *));
int      feof __P((FILE *));
int      ferror __P((FILE *));
int      fflush __P((FILE *));
int      fgetc __P((FILE *));
--
                (*(p)->_p = (c), (int)*(p)->_p++))
#endif

#define __sfeof(p)      (((p)->_flags & __SEOF) != 0)
#define __sferror(p)    (((p)->_flags & __SERR) != 0)
#define __sclearerr(p)  ((void)((p)->_flags &= ~(__SERR|__SEOF)))
#define __sfileno(p)    ((p)->_file)
#define __sfileno(p)    ((p)->_file)
--
 * See ISO/IEC 9945-1 ANSI/IEEE Std 1003.1 Second Edition 1996-07-12
 * B.8.2.7 for the rationale behind the *_unlocked() macros.
 */
#define feof_unlocked(p)        __sfeof(p)
#define ferror_unlocked(p)      __sferror(p)
#define clearerr_unlocked(p)    __sclearerr(p)

--
#endif

#ifndef  _THREAD_SAFE
#define feof(p)         feof_unlocked(p)
#define ferror(p)       ferror_unlocked(p)
#define clearerr(p)     clearerr_unlocked(p)

[newdhcpbox](0) $

--
nick black          "np:  the class of dashed hopes and idle dreams."

Failure to properly define _REENTRANT (and, where applicable, _THREAD_SAFE) will result in silent inclusions of unsafe, improperly-locked code and global variables in the place of thread-local data (ala errno(3)).

Linking objects with pthreads

On linux, this is as simple as things get: add -lpthread to the linker options. There's only two major pthread libraries on Linux, both of them affiliated with glibc -- LinuxThreads and then NPTL. NPTL is used on all recent distributions. -lpthread will link the primary pthreads implementation on your build machine.

On FreeBSD, the situation is a bit more complex, due to multiple pthread implementations, some of which are only available on recent FreeBSD versions. FreeBSD 4 provides the simplest case, with only two implementations:

• native BSD threads
• the LinuxThreads port (devel/linuxthreads; this does NOT require emulators/linux-base*)

all threaded code muse use the reentrant version of the gcc builtin libs; this is done via -lgcc_r (-pthread should set this up). In addition, use -llthread to link against LinuxThreads.

Use with autotools

The AX_PTHREAD macro from the Autoconf Archive is canonical, defining PTHREAD_LIBS, PTHREAD_CFLAGS, and PTHREAD_CC.