Theory and Practice of Sprixels - revision history

Dank: /* Implementation details */

2025-01-04T07:27:40Z

Implementation details

← Older revision		Revision as of 07:27, 4 January 2025
Line 128:		Line 128:
	The most complex expression of Sixel requires four phases: clear the old bitmaps (possibly overlapping with new bitmaps), draw any updated text that's visible through transparent areas of the bitmaps, draw the bitmaps, and draw any text (updated or not) that's visible above. If cuts are employed, this can be reduced to three phases, since the text above the bitmap can be printed below the bitmap. Kitty never requires more than two phases (update text, update bitmaps), since ordering isn't relevant in the Kitty protocol.		The most complex expression of Sixel requires four phases: clear the old bitmaps (possibly overlapping with new bitmaps), draw any updated text that's visible through transparent areas of the bitmaps, draw the bitmaps, and draw any text (updated or not) that's visible above. If cuts are employed, this can be reduced to three phases, since the text above the bitmap can be printed below the bitmap. Kitty never requires more than two phases (update text, update bitmaps), since ordering isn't relevant in the Kitty protocol.

	It's generally necessary to immediately follow a Kitty deletion command with the redraw, if it's being replaced. The animation extension coming in 0.20.0 ought improve significantly on this situation. Clearing a Sixel requires an overwrite with text or non-transparent sixels, or a screen clear/rectangular delete. In the case where a Sixel to be destroyed and a Sixel to be painted overlap, clearing the first sixel might require damaging cells underneath the second, hence the strong dependencies between the first three phases of Sixel rendering.		It's generally necessary to immediately follow a Kitty deletion command with the redraw, if it's being replaced. The animation extension coming in 0.20.0 ought improve significantly on this situation (note: it did). Clearing a Sixel requires an overwrite with text or non-transparent sixels, or a screen clear/rectangular delete. In the case where a Sixel to be destroyed and a Sixel to be painted overlap, clearing the first sixel might require damaging cells underneath the second, hence the strong dependencies between the first three phases of Sixel rendering.

	Some terminals can draw Sixels more quickly when the <tt>P2</tt> parameter is 0 than when it is 1 (foot explicitly makes this claim). We need 1 if we have transparent pixels, but we can use 0 otherwise. This can be derived directly from the TAM--if and only if all TAM cells are 0, we can set P2=0.		Some terminals can draw Sixels more quickly when the <tt>P2</tt> parameter is 0 than when it is 1 (foot explicitly makes this claim). We need 1 if we have transparent pixels, but we can use 0 otherwise. This can be derived directly from the TAM--if and only if all TAM cells are 0, we can set P2=0.

Dank: /* External Links */

2024-04-06T20:21:40Z

External Links

← Older revision		Revision as of 20:21, 6 April 2024
Line 145:		Line 145:
	* The [https://iterm2.com/documentation-images.html iTerm2 graphics protocol]		* The [https://iterm2.com/documentation-images.html iTerm2 graphics protocol]
	* Thomas Dickey's [https://invisible-island.net/xterm/ctlseqs/ctlseqs.html XTerm Control Sequences] document		* Thomas Dickey's [https://invisible-island.net/xterm/ctlseqs/ctlseqs.html XTerm Control Sequences] document
	* [https://gitlab.com/klamonte/jexer/blob/master/README.md Jexer], Autumn Lamonte's Java library inspired by Borland's "Turbo" line of ~~produts~~, is the only thing that comes close AFAIK		* [https://gitlab.com/klamonte/jexer/blob/master/README.md Jexer], Autumn Lamonte's Java library inspired by Borland's "Turbo" line of products, is the only thing that comes close AFAIK
	* [https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/12 Issue #12] of the terminal-ng working group		* [https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/12 Issue #12] of the terminal-ng working group
	* hackerb9's [https://github.com/hackerb9/vt340test vt340test] repository		* hackerb9's [https://github.com/hackerb9/vt340test vt340test] repository

Dank: /* Temporal vs Positional */

2024-04-06T20:20:46Z

Temporal vs Positional

← Older revision		Revision as of 20:20, 6 April 2024
Line 69:		Line 69:

	===Temporal vs Positional===		===Temporal vs Positional===
	Sixel writes the provided glyph as a unit, starting at the current cursor position (assuming we've entered DECSDM aka "Sixel Display Mode", which we always do—otherwise all sixels are emitted at the upper left corner of the display). Bits not specified in the sixel are transparent (assuming the Device Control String has used the value 1 for parameter <tt>P2</tt>), and transparent pixels leave undisturbed whatever was already present at the time of emission. Writing an entirely transparent region is a no-op. Writing a new character (or new Sixel pixels) anywhere in the graphic's region will wipe out the entirety of that cell. The Sixel data thus annihilated is not recoverable. This means that updating the text underneath a Sixel will blow the obscuring graphics away, requiring that the affected Sixel pixels be rerendered. In the event of stacked sprixels, we would potentially need to rerender the entire stack. In a naive implementation, changing one text cell "underneath" the octopus could result in painting 781 cells (the text cell actually being changed, and the 780 cells of the octopus). This is plainly undesirable.		Sixel writes the provided glyph as a unit, starting at the current cursor position (assuming we've entered DECSDM aka "Sixel Display Mode", which we always do—otherwise all sixels are emitted at the upper left corner of the display). Bits not specified in the sixel are transparent (assuming the Device Control String has used the value 1 for parameter <tt>P2</tt>), and transparent pixels leave undisturbed whatever was already present at the time of emission. Writing an entirely transparent region is a no-op. Writing a new character (or new Sixel pixels) anywhere in the graphic's region will wipe out the entirety of that cell. The Sixel data thus annihilated are not recoverable. This means that updating the text underneath a Sixel will blow the obscuring graphics away, requiring that the affected Sixel pixels be rerendered. In the event of stacked sprixels, we would potentially need to rerender the entire stack. In a naive implementation, changing one text cell "underneath" the octopus could result in painting 781 cells (the text cell actually being changed, and the 780 cells of the octopus). This is plainly undesirable.

	With Kitty, a graphic is positional. All text is assumed to be at z=0. Graphics with non-negative coordinates obscure text. Graphics with negative coordinates sit underneath text. Note that this means we must choose whether a Kitty sprixel is going to have text above it, or under it, but no arbitrary single sprixel can do both.		With Kitty, a graphic is positional. All text is assumed to be at z=0. Graphics with non-negative coordinates obscure text. Graphics with negative coordinates sit underneath text. Note that this means we must choose whether a Kitty sprixel is going to have text above it, or under it, but no arbitrary single sprixel can do both.

Dank: /* Color quantization */

2021-12-23T08:38:59Z

Color quantization

← Older revision		Revision as of 08:38, 23 December 2021
Line 85:		Line 85:
	* [https://en.wikipedia.org/wiki/Octree octrees], introduced by [https://sci-hub.do/10.1007/978-3-642-83492-9_20 Gervautz and Purgathofer] in 1988.		* [https://en.wikipedia.org/wiki/Octree octrees], introduced by [https://sci-hub.do/10.1007/978-3-642-83492-9_20 Gervautz and Purgathofer] in 1988.
	* Kohonen neural nets as introduced by [https://www.researchgate.net/profile/Anthony-Dekker/publication/232079905_Kohonen_neural_networks_for_optimal_colour_quantization/links/09e4150ba8346d8ce9000000/Kohonen-neural-networks-for-optimal-colour-quantization.pdf Dekker in NeuQuant]. Used by the [https://lib.rs/crates/color_quant color_quant crate].		* Kohonen neural nets as introduced by [https://www.researchgate.net/profile/Anthony-Dekker/publication/232079905_Kohonen_neural_networks_for_optimal_colour_quantization/links/09e4150ba8346d8ce9000000/Kohonen-neural-networks-for-optimal-colour-quantization.pdf Dekker in NeuQuant]. Used by the [https://lib.rs/crates/color_quant color_quant crate].
			** You'll also see [https://ieeexplore.ieee.org/document/939492 Hopfield nets]
	* K-means clustering, perhaps best known from [https://exoticorn.github.io/exoquant-rs/exoquant/ exoquant].		* K-means clustering, perhaps best known from [https://exoticorn.github.io/exoquant-rs/exoquant/ exoquant].

Dank: /* Color quantization */

2021-12-01T20:25:10Z

Color quantization

← Older revision		Revision as of 20:25, 1 December 2021
Line 77:		Line 77:

	==Color quantization==		==Color quantization==
	The Kitty protocol accepts arbitrary RGBA 32-bit words in the sRGB color space; our natural internal format is 24bpp RGB plus two bits of alpha, so that's great—our colorspaces match. Sixel is a palette-indexed format. Common sizes are 16-, 256-, and 1024-entry palettes (in XTerm, this is controlled with the <tt>numColorRegisters</tt> X resource). Palettes are independent unless the <tt>privateColorRegisters</tt> resource is ~~enabled~~). A color quantization step is thus necessary (note that this these quantizations can be performed in perfect parallelism when split across multiple frames/images, but not necessarily within a single frame). There are five essential color quantization algorithms:		The Kitty protocol accepts arbitrary RGBA 32-bit words in the sRGB color space; our natural internal format is 24bpp RGB plus two bits of alpha, so that's great—our colorspaces match. Sixel is a palette-indexed format. Common sizes are 16-, 256-, and 1024-entry palettes (in XTerm, this is controlled with the <tt>numColorRegisters</tt> X resource). Palettes are independent unless the <tt>privateColorRegisters</tt> resource is explicitly set false). A color quantization step is thus necessary (note that this these quantizations can be performed in perfect parallelism when split across multiple frames/images, but not necessarily within a single frame). There are five essential color quantization algorithms:

	[[File:Penrosetiling.png\|thumb\|200px\|right\|These images throw libsixel for a loop; <tt>img2sixel</tt> chugs along, and then generates no output.]]		[[File:Penrosetiling.png\|thumb\|200px\|right\|These images throw libsixel for a loop; <tt>img2sixel</tt> chugs along, and then generates no output.]]

Dank: /* Other people's bugs */

2021-12-01T20:23:12Z

Other people's bugs

← Older revision		Revision as of 20:23, 1 December 2021
Line 134:		Line 134:
	* XTerm through 367 doesn't update the pixel counts returned in the <tt>TIOCGWINSZ</tt> <tt>ioctl(2)</tt> when the font changes. I submitted a bug report for this, and it was [https://invisible-island.net/xterm/xterm.log.html#xterm_368 fixed in Patch #368].		* XTerm through 367 doesn't update the pixel counts returned in the <tt>TIOCGWINSZ</tt> <tt>ioctl(2)</tt> when the font changes. I submitted a bug report for this, and it was [https://invisible-island.net/xterm/xterm.log.html#xterm_368 fixed in Patch #368].
	* XTerm through at least 366 progressively slows down as Sixels are written in alternate mode, even if they are cleared. I submitted a bug report for this.		* XTerm through at least 366 progressively slows down as Sixels are written in alternate mode, even if they are cleared. I submitted a bug report for this.
	* XTerm through ~~at least 368~~ doesn't clear sixels with <tt>DECERA</tt>, <tt>DECFRA</tt>, etc. I [https://groups.google.com/g/notcurses/c/3ZBjA_v0EvQ submitted a patch for this], which was merged for [https://invisible-island.net/xterm/xterm.log.html#xterm_370 XTerm 370].		* XTerm through 369 doesn't clear sixels with <tt>DECERA</tt>, <tt>DECFRA</tt>, etc. I [https://groups.google.com/g/notcurses/c/3ZBjA_v0EvQ submitted a patch for this], which was merged for [https://invisible-island.net/xterm/xterm.log.html#xterm_370 XTerm 370].
	* Mlterm through at least 3.9.0 makes the cursor visible after emitting a Sixel, even if it was hidden before. I [https://github.com/arakiken/mlterm/issues/24 submitted a bug] for this.		* Mlterm through at least 3.9.0 makes the cursor visible after emitting a Sixel, even if it was hidden before. I [https://github.com/arakiken/mlterm/issues/24 submitted a bug] for this.
	* MLterm through 3.9.1 inverts the broadly-interpreted sense of DECSDM. I [https://github.com/arakiken/mlterm/pull/23 submitted a PR] for this, which was merged.		* MLterm through 3.9.1 inverts the broadly-interpreted sense of DECSDM. I [https://github.com/arakiken/mlterm/pull/23 submitted a PR] for this, which was merged.

Dank: /* Other people's bugs */

2021-11-18T20:54:11Z

Other people's bugs

← Older revision		Revision as of 20:54, 18 November 2021
Line 134:		Line 134:
	* XTerm through 367 doesn't update the pixel counts returned in the <tt>TIOCGWINSZ</tt> <tt>ioctl(2)</tt> when the font changes. I submitted a bug report for this, and it was [https://invisible-island.net/xterm/xterm.log.html#xterm_368 fixed in Patch #368].		* XTerm through 367 doesn't update the pixel counts returned in the <tt>TIOCGWINSZ</tt> <tt>ioctl(2)</tt> when the font changes. I submitted a bug report for this, and it was [https://invisible-island.net/xterm/xterm.log.html#xterm_368 fixed in Patch #368].
	* XTerm through at least 366 progressively slows down as Sixels are written in alternate mode, even if they are cleared. I submitted a bug report for this.		* XTerm through at least 366 progressively slows down as Sixels are written in alternate mode, even if they are cleared. I submitted a bug report for this.
	* XTerm through at least 368 doesn't clear sixels with <tt>DECERA</tt>, <tt>DECFRA</tt>, etc. I [https://groups.google.com/g/notcurses/c/3ZBjA_v0EvQ submitted a patch for this], ~~though it's probably better if the current behavior~~ [https://~~github~~.~~com~~/~~dankamongmen~~/~~notcurses/issues/1740~~#~~issuecomment-868964356 is retained~~].		* XTerm through at least 368 doesn't clear sixels with <tt>DECERA</tt>, <tt>DECFRA</tt>, etc. I [https://groups.google.com/g/notcurses/c/3ZBjA_v0EvQ submitted a patch for this], which was merged for [https://invisible-island.net/xterm/xterm.log.html#xterm_370 XTerm 370].
	* Mlterm through at least 3.9.0 makes the cursor visible after emitting a Sixel, even if it was hidden before. I [https://github.com/arakiken/mlterm/issues/24 submitted a bug] for this.		* Mlterm through at least 3.9.0 makes the cursor visible after emitting a Sixel, even if it was hidden before. I [https://github.com/arakiken/mlterm/issues/24 submitted a bug] for this.
	* MLterm through 3.9.1 inverts the broadly-interpreted sense of DECSDM. I [https://github.com/arakiken/mlterm/pull/23 submitted a PR] for this, which was merged.		* MLterm through 3.9.1 inverts the broadly-interpreted sense of DECSDM. I [https://github.com/arakiken/mlterm/pull/23 submitted a PR] for this, which was merged.

Dank: /* Sprixel flavors */

2021-10-28T18:02:55Z

Sprixel flavors

← Older revision		Revision as of 18:02, 28 October 2021
Line 12:		Line 12:
	** Implemented by at least XTerm, Mlterm, wezterm, and foot		** Implemented by at least XTerm, Mlterm, wezterm, and foot
	** with patches out for at least VTE, alacritty, and Windows Terminal.		** with patches out for at least VTE, alacritty, and Windows Terminal.
	* Kitty's protocol, advanced by Kovid Goyal.		** also supported by iTerm2, despite having its own protocol
			* Kitty's protocol, advanced by Kovid Goyal, and recently (Q3 2021) supported by wezterm.
	* The "1337" iTerm2 protocol, also implemented by wezterm.		* The "1337" iTerm2 protocol, also implemented by wezterm.
	* The [https://gitlab.com/klamonte/jexer/-/wikis/jexer-images Jexer Image Sequence] from Autumn Lamonte's Jexer		* The [https://gitlab.com/klamonte/jexer/-/wikis/jexer-images Jexer Image Sequence] from Autumn Lamonte's Jexer

Dank: /* External Links */ vt340test by hackerb9

2021-10-14T01:52:34Z

External Links: vt340test by hackerb9

← Older revision		Revision as of 01:52, 14 October 2021
Line 145:		Line 145:
	* [https://gitlab.com/klamonte/jexer/blob/master/README.md Jexer], Autumn Lamonte's Java library inspired by Borland's "Turbo" line of produts, is the only thing that comes close AFAIK		* [https://gitlab.com/klamonte/jexer/blob/master/README.md Jexer], Autumn Lamonte's Java library inspired by Borland's "Turbo" line of produts, is the only thing that comes close AFAIK
	* [https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/12 Issue #12] of the terminal-ng working group		* [https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/12 Issue #12] of the terminal-ng working group
			* hackerb9's [https://github.com/hackerb9/vt340test vt340test] repository

	'''previously: "[[Spooky_tmux_at_a_Distance\|spooky tmux at a distance]]" 2021-02-20'''		'''previously: "[[Spooky_tmux_at_a_Distance\|spooky tmux at a distance]]" 2021-02-20'''

Dank: /* Transparency and z-ordering */

2021-09-05T23:57:19Z

Transparency and z-ordering

← Older revision		Revision as of 23:57, 5 September 2021
Line 31:		Line 31:

	==Transparency and z-ordering==		==Transparency and z-ordering==
	These protocols differ from each other in fundamental, important ways, and forming an abstraction from them isn't trivial. Integrating them wholly into the z-ordered semantics of Notcurses was still more difficult. Notcurses works via <i>piles</i> (rendering contexts) of <i>planes</i> (drawing surfaces), totally ordered on a z axis. Higher planes blend with or obscure lower ones. When Notcurses <i>renders</i> a pile, it projects these planes down onto a single plane, solving a single surface; it <i>rasterizes</i> this virtual surface to the physical viewing area by encoding it as a set of terminal escape codes and UTF-8 data. Only cells which have changed need to be rasterized, and eliding unchanged cells is a critical optimization, usually cutting the output size down by 90% or more. Every output byte is processed several times before becoming visible (we must copy it to a kernel buffer, the terminal must read it from a kernel buffer, and the terminal must display the resulting cell), so trading some computation for fewer output bytes is well worth it.		These protocols differ from each other in fundamental, important ways, and forming an abstraction from them isn't trivial. Integrating them wholly into the z-ordered semantics of Notcurses was still more difficult. Notcurses works via <i>piles</i> (rendering contexts) of <i>planes</i> (drawing surfaces), with each pile's planes totally ordered on a z axis. Higher planes blend with or obscure lower ones. When Notcurses <i>renders</i> a pile, it projects these planes down onto a single plane, solving a single surface; it <i>rasterizes</i> this virtual surface to the physical viewing area by encoding it as a set of terminal escape codes and UTF-8 data. Only cells which have changed need to be rasterized, and eliding unchanged cells is a critical optimization, usually cutting the output size down by 90% or more. Every output byte is processed several times before becoming visible (we must copy it to a kernel buffer, the terminal must read it from a kernel buffer, and the terminal must display the resulting cell), so trading some computation for fewer output bytes is well worth it.

	The elegant model of planes of fixed-width, independent cells solving for a single cell matrix already breaks down in the presence of [[Unicode]] wide glyphs, perhaps foremost among them U+FDFD ARABIC LIGATURE BISMILLAH AR-RAHMAN AR-RAHEEM ﷽, a single glyph occupying anywhere from one to who-knows (I've seen 9) cells, depending on font, font engine, and terminal emulator. Wide characters cannot be printed in part, and printing another character anywhere atop a wide character obliterates all of the old character. Even restricting ourselves to characters two cells wide, a single character can annihilate four columns containing two characters:		The elegant model of planes of fixed-width, independent cells solving for a single cell matrix already breaks down in the presence of [[Unicode]] wide glyphs, perhaps foremost among them U+FDFD ARABIC LIGATURE BISMILLAH AR-RAHMAN AR-RAHEEM ﷽, a single glyph occupying anywhere from one to who-knows (I've seen 9) cells, depending on font, font engine, and terminal emulator. Wide characters cannot be printed in part, and printing another character anywhere atop a wide character obliterates all of the old character. Even restricting ourselves to characters two cells wide, a single character can annihilate four columns containing two characters: