Check out my first novel, midnight's simulacra!

Hackery: Difference between revisions

From dankwiki
 
(46 intermediate revisions by the same user not shown)
Line 1: Line 1:
==Open source detritus==
==Open source detrital TODOs==
* speed up update-mandb
* fix up ext3grep to work on ext4 filesystems
* fix up ext3grep to work on ext4 filesystems
* Implement handling of [[C99]] [http://gcc.gnu.org/onlinedocs/gcc/Variadic-Macros.html variadic macros] in [http://www.splint.org/ splint] (see [http://www.advogato.org/person/muks/diary.html/start=0 this], [http://www.buzztard.org/index.php/Splint this] and [http://www.splint.org/faq.html#quest18b of course this])
* Implement handling of [[C99]] [http://gcc.gnu.org/onlinedocs/gcc/Variadic-Macros.html variadic macros] in [http://www.splint.org/ splint] (see [http://www.advogato.org/person/muks/diary.html/start=0 this], [http://www.buzztard.org/index.php/Splint this] and [http://www.splint.org/faq.html#quest18b of course this])
Line 7: Line 8:
* "taking [[OpenSSL|OpenSSL's]] <tt>RAND_bytes()</tt> (which uses <tt>/dev/urandom</tt>, EGD, or a pregenerated seed file) function and stirring that into a target buffer of arbitrary length, such that use of actual high-quality entropy bits is constant (probably a read of several words each time a thread calls into (threadsafe from the start, no *_r() crap!) <tt>FAUXRAND_bytes()</tt> for the first time) has been on my plate since...well, since just now." (mail to Dr. Richard Vuduc, 2009-09-19)
* "taking [[OpenSSL|OpenSSL's]] <tt>RAND_bytes()</tt> (which uses <tt>/dev/urandom</tt>, EGD, or a pregenerated seed file) function and stirring that into a target buffer of arbitrary length, such that use of actual high-quality entropy bits is constant (probably a read of several words each time a thread calls into (threadsafe from the start, no *_r() crap!) <tt>FAUXRAND_bytes()</tt> for the first time) has been on my plate since...well, since just now." (mail to Dr. Richard Vuduc, 2009-09-19)
* [[mpd]] ought use filesystem change notification events to trigger database changes, not periodic or manual rescans
* [[mpd]] ought use filesystem change notification events to trigger database changes, not periodic or manual rescans
* [[valgrind]] is missing some obscure ioctl's, including [[ethtool|ethtool's]]
* canScan
* systemd Restart-with-oneshot fix
* use [[Outcurses]] with usbtop for ncurses mode
* fix bug in usbtop where disappeared devices remain forever
* systemd system-wide service for pulseaudio (requested on freedesktop.org wiki!)
* cuda fft for various SDR
* step-cli -- update debian stuff, make it bizzuild
* ubertooth-dfu without `-d` just exits cleanly, doing nothing
* neat fpga project -- bluetooth 5 (2mbit ble channel) 40x channel sniffer
===Various open source contributions (very incomplete)===
===Various open source contributions (very incomplete)===
* doctest:
** [https://github.com/onqtam/doctest/pull/403 properly align] summary table even for large values
* Kitty:
** [https://github.com/kovidgoyal/kitty/pull/3105 fixed] a bug in sextant drawing
* [[CMake]]:
** [https://gitlab.kitware.com/cmake/cmake/merge_requests/3845 fixed] up the Curses module
* [https://github.com/OpenShot/libopenshot OpenShot]:
** [https://github.com/OpenShot/libopenshot/commit/4a1d133da85e529b158f9a34518c41feb150d71c fixed] a compiler warning
* libqmi:
** added support for [https://gitlab.freedesktop.org/mobile-broadband/libqmi/issues/10 device specification using symlinks]
* Alpine Linux:
** added parsechangelog tools to dpkg
** packaged capnproto and nlohmann-json
* Compiz:
** fixed [https://github.com/compiz-reloaded/compiz/commit/8884ea41b72a96ecc8bf55029033af48d3d3fb97 test for decorator specification]
* iperf2:
** [https://sourceforge.net/p/iperf2/discussion/general/thread/b27f2bc6 Fixed] bad delete leading to memory corruption
* [[Ncurses]]:
** [http://comments.gmane.org/gmane.comp.lib.ncurses.bugs/4910 Fixed] COLOR_PAIR() and PAIR_NUMBER() macros
* [[Valgrind]]:
** Added support for [https://bugs.kde.org/show_bug.cgi?id=302827 CDROM_GET_CAPABILITY] <tt>ioctl</tt>
** Added support for [https://bugs.kde.org/show_bug.cgi?id=410556 BLKID] <tt>ioctls</tt>
* libblkid (util-linux)
** [https://github.com/karelzak/util-linux/commit/ffab21e12846dd9b9403c881721e415493805bd1 Fix #1]
** [https://github.com/karelzak/util-linux/commit/330ff7edc480b4e897f946c30a2afa38c916b9d0 Fix #2]
** [https://github.com/karelzak/util-linux/commit/d8a5b55012c2e01bff78a9c1237f4f2a93bd9c3a Fix #3]
** [https://github.com/karelzak/util-linux/commit/28a47f13d9c33c97279742d83687bff4d79922f4 Fix #4]
* [http://www.wireshark.org/about.html Wireshark]:
* [http://www.wireshark.org/about.html Wireshark]:
** [http://www.ethereal.com/lists/ethereal-users/200402/msg00153.html Fixed handling] of IPv4 fragmentation bits.
** [https://www.wireshark.org/lists/ethereal-dev/200402/msg00512.html Fixed handling] of IPv4 fragmentation bits.
* [http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 iproute2]
** Many [http://git.kernel.org/?p=linux/kernel/git/shemminger/iproute2.git;a=blobdiff;f=man/man8/ip.8;h=0f9f454737c5a5977026752321d620d7bee79158;hp=68887bacff00e6e03f4e035a5935f557aa79bd83;hb=db4a7f198b6193a1e584c3b6647d92bb6c66fb52;hpb=14a1c164d12e32415acc44ef566fcf52ff4dd113 man page] fixes
* [[Linux APIs|Linux kernel]]:
* [[Linux APIs|Linux kernel]]:
** [https://patchwork.kernel.org/patch/11039443/ Whitelisted] the [[Lenovo]] T580's synaptics touchpad for SMBus
** [http://lkml.org/lkml/2010/5/4/6 Added support] for Model-30 [[Nehalem]] processors in [[Oprofile]] ([http://marc.info/?l=linux-kernel&m=127294830417492&w=2 more discussion] on oprofile-users)
** [http://lkml.org/lkml/2010/5/4/6 Added support] for Model-30 [[Nehalem]] processors in [[Oprofile]] ([http://marc.info/?l=linux-kernel&m=127294830417492&w=2 more discussion] on oprofile-users)
** [http://lkml.indiana.edu/hypermail/linux/kernel/0906.3/02414.html Addressed issues] in [[pages|HugeTLBFS]]'s API.
** [http://lkml.indiana.edu/hypermail/linux/kernel/0906.3/02414.html Addressed issues] in [[pages|HugeTLBFS]]'s API.
** [http://lkml.indiana.edu/hypermail/linux/net/0301.1/0037.html Addressed] an issue in IPv4 address/route removal.
** [http://lkml.indiana.edu/hypermail/linux/net/0301.1/0037.html Addressed] an issue in IPv4 address/route removal.
** Worked out some kinks in Matrox G400 framebuffer and SiS-5591 IDE drivers.
** Worked out some kinks in Matrox G400 framebuffer and SiS-5591 IDE drivers.
** [https://fossies.org/linux/privat/old/ngpt-2.2.1.tar.gz:a/ngpt-2.2.1/THANKS Next Generation Posix Threading] work + maintenance
* [http://sourceforge.net/projects/strace/ strace]:
* [http://sourceforge.net/projects/strace/ strace]:
** [http://www.mail-archive.com/strace-devel@lists.sourceforge.net/msg00614.html Addressed issues] in various architectures' [[Linux APIs|sendfile(2)]] support.
** [http://www.mail-archive.com/strace-devel@lists.sourceforge.net/msg00614.html Addressed issues] in various architectures' [[Linux APIs|sendfile(2)]] support.
Line 20: Line 63:
** [http://git.choralone.org/?p=x86info.git;a=commit;h=17d9bf43f775f5a780bcccbca52e2ce37c3ca1f1 fixed] feature-specific TLB and cache detection
** [http://git.choralone.org/?p=x86info.git;a=commit;h=17d9bf43f775f5a780bcccbca52e2ce37c3ca1f1 fixed] feature-specific TLB and cache detection
** [http://git.choralone.org/?p=x86info.git;a=commit;h=a8af3758826f545b289a40ecefa627de1e5e9e90 added] support for several Intel caches
** [http://git.choralone.org/?p=x86info.git;a=commit;h=a8af3758826f545b289a40ecefa627de1e5e9e90 added] support for several Intel caches
* [http://dynamorio.org/ DynamoRIO]:
** [http://groups.google.com/group/dynamorio-users/browse_thread/thread/72dd27ca8f5ead66/5064e2e1d98b8fcf?lnk=gst&q=nick#5064e2e1d98b8fcf added] [[Nehalem]] support
* avant-wireless
** An [http://wiki.awn-project.org/Applet_Gallery AWN applet] which tracks wireless connection properties, and spawns wpa_gui
* iw
** [http://thread.gmane.org/gmane.linux.kernel.wireless.general/82070 Correct unsigned] ints used to hold signed data
* gnome-session
** [https://bugzilla.gnome.org/show_bug.cgi?id=689009 Fixed] gnome-session-properties man page
* Snort
** Various [http://copilotco.com/mail-archives/snort-users.2003/msg06337.html signature documentation]


==Projects with their own pages==
==Projects with their own pages==
* [[ptracer]]: Quick-n-dirty instruction trace generation tool
* [[libdank]]: Long-term personal collection of routines / application frameworks with a low-level feel
* [[libdank]]: Long-term personal collection of routines / application frameworks with a low-level feel
* [[TANGE]]: Terminal Application (Next-Generation Emulation)
* [[TANGE]]: Terminal Application (Next-Generation Emulation)
Line 29: Line 83:
* [[libtorque]]: Multithreaded event handler for UNIX on manycore [[NUMA]]
* [[libtorque]]: Multithreaded event handler for UNIX on manycore [[NUMA]]
* [[daytripper]]: Binary translation to take advantage of Intel's Loop Stream Detector
* [[daytripper]]: Binary translation to take advantage of Intel's Loop Stream Detector
* [[CUBAR]], a collection of tools for testing [[CUDA]]'s security model
* [[CUBAR]]: collection of tools for testing [[CUDA]]'s security model
* [[libcudest]], an open-source implementation of the [[CUDA]] userspace
* [[libcudest]]: open-source implementation of the [[CUDA]] userspace
* [[LRUmap]], O(1) LRU for massive numbers of sets
* [[LRUmap]]: O(1) LRU for massive numbers of sets
* [[omphalos]]: multi-pronged network discovery
* [[growlight]]: multifaceted disk/adapter tool and system installer for [[SprezzOS]]
* [[Xcurses]]: a SYSV [[ncurses|curses]] implementation making direct use of [[X]]-graphics primitives
* [[Outcurses]]: a high-level UI library atop [[ncurses]]
[[CATEGORY: Networking]]
* Various projects on [http://github.com/dankamongmen GitHub] which I've not yet documented here...
* Various projects on [http://github.com/dankamongmen GitHub] which I've not yet documented here...


Line 61: Line 120:
** To be paired with vulnerability correlation, active probing, histories (ala SourceFire RNA?)
** To be paired with vulnerability correlation, active probing, histories (ala SourceFire RNA?)
* Higher layers do not restrict the inductive analysis of lower layers, but influence the deduction
* Higher layers do not restrict the inductive analysis of lower layers, but influence the deduction
* Automata-based analysis of content builds up possibility space of what it CAN be
* [[Automata|Automata-based]] analysis of content builds up possibility space of what it CAN be
** Probability multipliers (products of series) paired with non-deterministic automata
** Probability multipliers (products of series) paired with [[Automata|non-deterministic automata]]
* Expert knowledge-based reduction of possibility space adds input as to what it OUGHT be
* Expert knowledge-based reduction of possibility space adds input as to what it OUGHT be
* Neural net-based learning with state determines what it IS
* Neural net-based learning with state determines what it IS
** Over time, confidence in matching both recurring and new traffic increases
** Over time, confidence in matching both recurring and new traffic increases
* Uses [[Research ideas#Parvenu|Parvenu]] as a pattern-matching helper
* Uses [[Research ideas#Parvenu|Parvenu]] as a pattern-matching helper
* Forms an analysis engine for [[Omphalos]]


===liburine===
===liburine===
Line 79: Line 139:
* Checks sysctls, <tt>netstat</tt>, <tt>/proc</tt>, etc for network settings. Makes extensive sensible recommendations.
* Checks sysctls, <tt>netstat</tt>, <tt>/proc</tt>, etc for network settings. Makes extensive sensible recommendations.
** Is device polling / NAPI in use? Do firewall rules disallow PMTU discovery?
** Is device polling / NAPI in use? Do firewall rules disallow PMTU discovery?
** Is TCP FRTO being used in the presence of wireless links? etc
** Is [[TCP]] FRTO being used in the presence of wireless links? etc
* Analyzes pcaps or raw sockets ''in situ'', with filters, performing detailed diagnostics ie
* Analyzes pcaps or raw sockets ''in situ'', with filters, performing detailed diagnostics ie
** "TCP 1323 high-performance extensions are being used in 24% of the connections"
** "[[TCP]] 1323 high-performance extensions are being used in 24% of the connections"
** "Fragmentation is resulting from port 7000's large UDP emissions, can we tune it this way..."
** "Fragmentation is resulting from port 7000's large UDP emissions, can we tune it this way..."
** "[http://www.youtube.com/watch?v=i6w9YHfabUU Make an incision, Doctor Limpf.] I'm going to massage the heart."


==Compilation/binaries==
==Compilation/binaries==
Line 96: Line 157:
** maybe something to test combinations of command line parameters, as well?
** maybe something to test combinations of command line parameters, as well?
* getgccopt -- discover the compiler options used to build an object. difficult!
* getgccopt -- discover the compiler options used to build an object. difficult!
* arrlimiter -- run object code with various rlimit restrictions to test failure paths
* arrlimiter -- run object code with various [[rlimit]] restrictions to test failure paths
* ploom -- pthreads debugging from process (tracing framework, object lookup + pthread knowledge, contention measures, etc)
* ploom -- [[pthreads]] debugging from process (tracing framework, object lookup + pthread knowledge, contention measures, etc)


==Aborted/abandoned projects (very incomplete)==
==Aborted/abandoned projects (very incomplete)==

Latest revision as of 21:44, 5 May 2023

Open source detrital TODOs

  • speed up update-mandb
  • fix up ext3grep to work on ext4 filesystems
  • Implement handling of C99 variadic macros in splint (see this, this and of course this)
  • Review cryptographic implementations in pidgin -- last time I looked (back in the bad ol' gaim days), its usage of OpenSSL was all FUBAR
  • Subversion's --xml and --ignore-externals options couldn't be used together properly in svn status, at some point. Investigate, rectify.
  • Various Debian-related things
  • "taking OpenSSL's RAND_bytes() (which uses /dev/urandom, EGD, or a pregenerated seed file) function and stirring that into a target buffer of arbitrary length, such that use of actual high-quality entropy bits is constant (probably a read of several words each time a thread calls into (threadsafe from the start, no *_r() crap!) FAUXRAND_bytes() for the first time) has been on my plate since...well, since just now." (mail to Dr. Richard Vuduc, 2009-09-19)
  • mpd ought use filesystem change notification events to trigger database changes, not periodic or manual rescans
  • valgrind is missing some obscure ioctl's, including ethtool's
  • canScan
  • systemd Restart-with-oneshot fix
  • use Outcurses with usbtop for ncurses mode
  • fix bug in usbtop where disappeared devices remain forever
  • systemd system-wide service for pulseaudio (requested on freedesktop.org wiki!)
  • cuda fft for various SDR
  • step-cli -- update debian stuff, make it bizzuild
  • ubertooth-dfu without `-d` just exits cleanly, doing nothing
  • neat fpga project -- bluetooth 5 (2mbit ble channel) 40x channel sniffer

Various open source contributions (very incomplete)

Projects with their own pages

  • ptracer: Quick-n-dirty instruction trace generation tool
  • libdank: Long-term personal collection of routines / application frameworks with a low-level feel
  • TANGE: Terminal Application (Next-Generation Emulation)
  • ctxdiff: Context-sensitive fractal fuzzy diffing (not at all like shingleprinting)
  • xsh: Exactly what it sounds like!
  • makelint, which immediately suggests? lintmake
  • libtorque: Multithreaded event handler for UNIX on manycore NUMA
  • daytripper: Binary translation to take advantage of Intel's Loop Stream Detector
  • CUBAR: collection of tools for testing CUDA's security model
  • libcudest: open-source implementation of the CUDA userspace
  • LRUmap: O(1) LRU for massive numbers of sets
  • omphalos: multi-pronged network discovery
  • growlight: multifaceted disk/adapter tool and system installer for SprezzOS
  • Xcurses: a SYSV curses implementation making direct use of X-graphics primitives
  • Outcurses: a high-level UI library atop ncurses
  • Various projects on GitHub which I've not yet documented here...

Core stuff

Parvenu

  • One string-matching automaton to Rule them All! It must handle:
    • Thousands (millions?) of patterns concurrently
    • Initially targeting GigE wire speeds
    • UTF-8 (at a minimum) and various transcodings
    • Small-memory (embedded) environments
    • Most elements of regular expressions (definitely all the syntactic sugar)
  • Uses libblaze as a memory management helper

libblaze

  • Use cpuid to select and dlopen(2) a processor-specific set of low-level routines
  • Expose cpuid and SMP details
  • Expose algorithms for memory management / layout tuned to cache/DRAM parameters (detected with SPD)
  • Material largely inspired by Warren's Hacker's Delight.
  • Might have already been superseded by liboil. That damn open source community moves fast!

Networking

Zetetic

  • Network traffic analysis (ala Wireshark or tcpdump) with strong covert channel detection
  • Passive network modeling (protocols, services, hosts, versions) initially
    • To be paired with vulnerability correlation, active probing, histories (ala SourceFire RNA?)
  • Higher layers do not restrict the inductive analysis of lower layers, but influence the deduction
  • Automata-based analysis of content builds up possibility space of what it CAN be
  • Expert knowledge-based reduction of possibility space adds input as to what it OUGHT be
  • Neural net-based learning with state determines what it IS
    • Over time, confidence in matching both recurring and new traffic increases
  • Uses Parvenu as a pattern-matching helper
  • Forms an analysis engine for Omphalos

liburine

  • The inverse of Zetetic. Take a leak (alternatively: "You're in")!
  • Use arbitrary combinations of protocols (and fuzzing/embedding thereof) to find channels
  • Goals: if even a bit of controllable information can be used as a channel
  • Goals: Zetetic should be able to find us, but nothing of lesser power!
  • The ultimate assistant for: whistleblowers, samizdat publishers, hax0rs stuck in airports, Iranians

drbenway

  • Checks sysctls, netstat, /proc, etc for network settings. Makes extensive sensible recommendations.
    • Is device polling / NAPI in use? Do firewall rules disallow PMTU discovery?
    • Is TCP FRTO being used in the presence of wireless links? etc
  • Analyzes pcaps or raw sockets in situ, with filters, performing detailed diagnostics ie
    • "TCP 1323 high-performance extensions are being used in 24% of the connections"
    • "Fragmentation is resulting from port 7000's large UDP emissions, can we tune it this way..."
    • "Make an incision, Doctor Limpf. I'm going to massage the heart."

Compilation/binaries

gcc stuff

debugging / reverse engineering tools

  • getgetopt -- discover a program's command line parameters via object analysis
    • maybe something to test combinations of command line parameters, as well?
  • getgccopt -- discover the compiler options used to build an object. difficult!
  • arrlimiter -- run object code with various rlimit restrictions to test failure paths
  • ploom -- pthreads debugging from process (tracing framework, object lookup + pthread knowledge, contention measures, etc)

Aborted/abandoned projects (very incomplete)

  • Investigate suffer and see whether it's feasible (update: fbcmd hendels its dendels Dank 23:05, 8 December 2008 (UTC))
  • Gyre, a programming language.
Retrieved from "https://nick-black.com/dankwiki/index.php?title=Hackery&oldid=9857"