CUBAR

(19 intermediate revisions by the same user not shown)

Line 1:

[[File:TheG8.png|right|thumb|CUDA'd-up beyond all repair]]

[[CUDA]] (and [http://en.wikipedia.org/wiki/GPGPU General-Purpose Graphics Processing Unit] programming in general) is rapidly becoming a mainstay of high-performance computing. As CUDA and OpenCL move off of the workstation, and into the server -- off of the console, and into the cluster -- the security of these systems will become critical parts of the associated [http://en.wikipedia.org/wiki/Trusted_computing_base trusted computing base]. Even ignoring the issue of multiuser security, the properties of isolation and (to a lesser extent) confidentiality are important for debugging, profiling and reproducibility. I've authored the ~~<s>cudafucker</s>~~ CUBAR set of tools to investigate the security properties -- primarily the means and parameters of memory protection, and the division of protection between soft- and hardware -- of [[CUDA]] on NVIDIA hardware since the G80 architecture.

[[CUDA]] (and [http://en.wikipedia.org/wiki/GPGPU General-Purpose Graphics Processing Unit] programming in general) is rapidly becoming a mainstay of high-performance computing. As CUDA and OpenCL move off of the workstation, and into the server -- off of the console, and into the cluster -- the security of these systems will become critical parts of the associated [http://en.wikipedia.org/wiki/Trusted_computing_base trusted computing base]. Even ignoring the issue of multiuser security, the properties of isolation and (to a lesser extent) confidentiality are important for debugging, profiling and reproducibility. I've authored the CUBAR set of tools to investigate the security properties -- primarily the means and parameters of memory protection, and the division of protection between soft- and hardware -- of [[CUDA]] on NVIDIA hardware since the G80 architecture. This was essential information in the attempt to implement an open source CUDA software stack (see [[libcudest]]).

See our 2010 paper, "[[Media:Cubar2010.pdf|My Other Computer is Your GPU]]."

==Questions==

===Memory details===

Line 38:

Line 39:

===Protection===

* What mechanisms, if any, exist to protect memory? At what granularities (of address and access) do they operate?

** What about registers? CUDA doesn't do traditional context switches due to register preallocation.

* How is memory protection split across hardware, kernelspace, and userspace?

** Any userspace protection can, of course, be trivially subverted

Line 44:

Line 46:

* What memories, if any, are scrubbed between kernels' execution?

* How many different regions can be tracked? How many contexts? What behavior exists at these limits?

===Variation===

* Have these mechanisms changed over various hardware?

Line 72:

Line 75:

==Code==

* GitHub project: [http://github.com/dankamongmen/~~wdp/tree/master/cs8803ss-project/~~ dankamongmen:~~wdp/cs8803ss-project~~]

* GitHub project: [http://github.com/dankamongmen/cubar dankamongmen:cubar]

===Building===

Clone the [[git]] repository. Define (either via shell export or within a new file Makefile.local):

Line 79:

Line 82:

==Tools==

[[File:Cudash-hyperboxl.png|right|thumb|cudash in use]]

===CUBAR tools===

Only those tools which launch CUDA kernels use the Runtime API. They might be converted to the Driver API.

===Driver API===

====Driver API====

* <tt>cudaminimal</tt> - The minimal possible CUDA program (a call to [http://developer.download.nvidia.com/compute/cuda/3_0/toolkit/docs/online/group__CUINIT_g4703189f4c7f490c73f77942a3fa8443.html#g4703189f4c7f490c73f77942a3fa8443 cuInit()]), used for <tt>strace(1)</tt> etc.

* <tt>cudapinner</tt> - Allocates and pins (locks) all possible shared (system) memory, using the minimum necessary number of distinct allocations.

* <tt>cudaspawner</tt> - Determine the maximum number of CUcontexts supported by a device, optionally applying memory pressure proportional to ''n''.

* <tt>cudastuffer</tt> - Allocates all possible memory on a device, using the minimum necessary number of distinct allocations.

====Runtime API====

===Runtime API===

* <tt>cudadump</tt> - Driver for <tt>cudaranger</tt> and sanity checker ('''FIXME: ought be split up into two or more tools'''). For each device, it:

** verifies that the constant memory region can be read in its entirety by a CUDA kernel

Line 91:

Line 96:

* <tt>cudaranger</tt> - Attempts to perform selectable memory accesses across the specified virtual address range of a device. Calculates the number of non-zero words.

It is necessary that <tt>cudadump</tt> spawn <tt>cudaranger</tt> rather than launching the kernels itself; a kernel failure results in all libcuda calls returning an early error (CUDA_ERROR_DEINITIALIZED (4) or CUDA_ERROR_LAUNCH_FAILURE (700)) for the remainder of a process's life (ie, creating a new context usually also fails '''FIXME: provide proof''').

====cudash====

The CUDA Shell (currently making use of the runtime api for kernel launches) facilitates stateful, invasive examination of the CUDA memory/process system. It has online help via the "help" command.

* ''FIXME'': document commands

===Other tools===

* [http://nouveau.freedesktop.org/wiki/REnouveau REnouveau], the Reverse-Engineering-for-nouveau tool

* [http://nouveau.freedesktop.org/wiki/MmioTrace MMIOtrace], in-kernel since 2.6.27

* The x86 [[Page Attribute Tables]] and [[MTRR]]s can be examined; see those pages for more detail

* [[gdb]] and NVIDIA's <tt>cuda-gdb</tt>

* [http://wiki.github.com/laanwj/decuda/ van der Laan's] CUBIN assembler and disassembler, <tt>decuda</tt> and <tt>cudasm</tt>

* [http://nouveau.freedesktop.org/wiki/Nvtrace nvtrace] from the nouveau project

* [http://0x04.net/cgit/index.cgi/nv50dis/ nv50dis], a NV50 disassembler

* [http://code.google.com/p/gpuocelot/ gpuocelot], a PTX binary translator targeting CUDA and multithreaded x86

* [http://upcommons.upc.edu/pfc/bitstream/2099.1/7589/1/PFC.pdf libecuda], a CUDART replacement and PTX emulator

==See Also==

Line 96:

Line 114:

** but that's [http://www.eecs.berkeley.edu/~demmel/ James Demmel] (aka a Real American Badass) for you!

* "[http://www.eecg.toronto.edu/~myrto/gpuarch-ispass2010.pdf Demystifying the GPU Through Benchmarking]" continues with this theme.

* LWN's State of Nouveau articles ([http://lwn.net/Articles/269558 Part 1] and [http://lwn.net/Articles/270830/ Part 2])

* The [http://nouveau.freedesktop.org/wiki/IrcChatLogs Nouveau Development Companion]

* This Japanese [http://pc.watch.impress.co.jp/docs/2008/0617/kaigai446.htm article] has some great diagrams

* [[libcudest]], my effort to reverse engineer and reimplement libcuda.so

[[CATEGORY: GPGPU]]

[[CATEGORY: Projects]]

CUBAR: Difference between revisions

Dank (talk | contribs)

@@ Line 1: / Line 1: @@
 [[File:TheG8.png|right|thumb|CUDA'd-up beyond all repair]]
-[[CUDA]] (and [http://en.wikipedia.org/wiki/GPGPU General-Purpose Graphics Processing Unit] programming in general) is rapidly becoming a mainstay of high-performance computing. As CUDA and OpenCL move off of the workstation, and into the server -- off of the console, and into the cluster -- the security of these systems will become critical parts of the associated [http://en.wikipedia.org/wiki/Trusted_computing_base trusted computing base]. Even ignoring the issue of multiuser security, the properties of isolation and (to a lesser extent) confidentiality are important for debugging, profiling and reproducibility. I've authored the <s>cudafucker</s> CUBAR set of tools to investigate the security properties -- primarily the means and parameters of memory protection, and the division of protection between soft- and hardware -- of [[CUDA]] on NVIDIA hardware since the G80 architecture.
+[[CUDA]] (and [http://en.wikipedia.org/wiki/GPGPU General-Purpose Graphics Processing Unit] programming in general) is rapidly becoming a mainstay of high-performance computing. As CUDA and OpenCL move off of the workstation, and into the server -- off of the console, and into the cluster -- the security of these systems will become critical parts of the associated [http://en.wikipedia.org/wiki/Trusted_computing_base trusted computing base]. Even ignoring the issue of multiuser security, the properties of isolation and (to a lesser extent) confidentiality are important for debugging, profiling and reproducibility. I've authored the CUBAR set of tools to investigate the security properties -- primarily the means and parameters of memory protection, and the division of protection between soft- and hardware -- of [[CUDA]] on NVIDIA hardware since the G80 architecture. This was essential information in the attempt to implement an open source CUDA software stack (see [[libcudest]]).
+See our 2010 paper, "[[Media:Cubar2010.pdf|My Other Computer is Your GPU]]."
 ==Questions==
 ===Memory details===
@@ Line 38: / Line 39: @@
 ===Protection===
 * What mechanisms, if any, exist to protect memory? At what granularities (of address and access) do they operate?
+** What about registers? CUDA doesn't do traditional context switches due to register preallocation.
 * How is memory protection split across hardware, kernelspace, and userspace?
 ** Any userspace protection can, of course, be trivially subverted
@@ Line 44: / Line 46: @@
 * What memories, if any, are scrubbed between kernels' execution?
 * How many different regions can be tracked? How many contexts? What behavior exists at these limits?
 ===Variation===
 * Have these mechanisms changed over various hardware?
@@ Line 72: / Line 75: @@
 ==Code==
-* GitHub project: [http://github.com/dankamongmen/wdp/tree/master/cs8803ss-project/ dankamongmen:wdp/cs8803ss-project]
+* GitHub project: [http://github.com/dankamongmen/cubar dankamongmen:cubar]
 ===Building===
 Clone the [[git]] repository. Define (either via shell export or within a new file Makefile.local):
@@ Line 79: / Line 82: @@
 ==Tools==
+[[File:Cudash-hyperboxl.png|right|thumb|cudash in use]]
+===CUBAR tools===
 Only those tools which launch CUDA kernels use the Runtime API. They might be converted to the Driver API.
-===Driver API===
+====Driver API====
 * <tt>cudaminimal</tt> - The minimal possible CUDA program (a call to [http://developer.download.nvidia.com/compute/cuda/3_0/toolkit/docs/online/group__CUINIT_g4703189f4c7f490c73f77942a3fa8443.html#g4703189f4c7f490c73f77942a3fa8443 cuInit()]), used for <tt>strace(1)</tt> etc.
+* <tt>cudapinner</tt> - Allocates and pins (locks) all possible shared (system) memory, using the minimum necessary number of distinct allocations.
 * <tt>cudaspawner</tt> - Determine the maximum number of CUcontexts supported by a device, optionally applying memory pressure proportional to ''n''.
 * <tt>cudastuffer</tt> - Allocates all possible memory on a device, using the minimum necessary number of distinct allocations.
+====Runtime API====
-===Runtime API===
 * <tt>cudadump</tt> - Driver for <tt>cudaranger</tt> and sanity checker ('''FIXME: ought be split up into two or more tools'''). For each device, it:
 ** verifies that the constant memory region can be read in its entirety by a CUDA kernel
@@ Line 91: / Line 96: @@
 * <tt>cudaranger</tt> - Attempts to perform selectable memory accesses across the specified virtual address range of a device. Calculates the number of non-zero words.
 It is necessary that <tt>cudadump</tt> spawn <tt>cudaranger</tt> rather than launching the kernels itself; a kernel failure results in all libcuda calls returning an early error (CUDA_ERROR_DEINITIALIZED (4) or CUDA_ERROR_LAUNCH_FAILURE (700)) for the remainder of a process's life (ie, creating a new context usually also fails '''FIXME: provide proof''').
+====cudash====
+The CUDA Shell (currently making use of the runtime api for kernel launches) facilitates stateful, invasive examination of the CUDA memory/process system. It has online help via the "help" command.
+* ''FIXME'': document commands
+===Other tools===
+* [http://nouveau.freedesktop.org/wiki/REnouveau REnouveau], the Reverse-Engineering-for-nouveau tool
+* [http://nouveau.freedesktop.org/wiki/MmioTrace MMIOtrace], in-kernel since 2.6.27
+* The x86 [[Page Attribute Tables]] and [[MTRR]]s can be examined; see those pages for more detail
+* [[gdb]] and NVIDIA's <tt>cuda-gdb</tt>
+* [http://wiki.github.com/laanwj/decuda/ van der Laan's] CUBIN assembler and disassembler, <tt>decuda</tt> and <tt>cudasm</tt>
+* [http://nouveau.freedesktop.org/wiki/Nvtrace nvtrace] from the nouveau project
+* [http://0x04.net/cgit/index.cgi/nv50dis/ nv50dis], a NV50 disassembler
+* [http://code.google.com/p/gpuocelot/ gpuocelot], a PTX binary translator targeting CUDA and multithreaded x86
+* [http://upcommons.upc.edu/pfc/bitstream/2099.1/7589/1/PFC.pdf libecuda], a CUDART replacement and PTX emulator
 ==See Also==
@@ Line 96: / Line 114: @@
 ** but that's [http://www.eecs.berkeley.edu/~demmel/ James Demmel] (aka a Real American Badass) for you!
 * "[http://www.eecg.toronto.edu/~myrto/gpuarch-ispass2010.pdf Demystifying the GPU Through Benchmarking]" continues with this theme.
+* LWN's State of Nouveau articles ([http://lwn.net/Articles/269558 Part 1] and [http://lwn.net/Articles/270830/ Part 2])
+* The [http://nouveau.freedesktop.org/wiki/IrcChatLogs Nouveau Development Companion]
+* This Japanese [http://pc.watch.impress.co.jp/docs/2008/0617/kaigai446.htm article] has some great diagrams
+* [[libcudest]], my effort to reverse engineer and reimplement libcuda.so
+[[CATEGORY: GPGPU]]
+[[CATEGORY: Projects]]