Theory and Practice of Sprixels: Difference between revisions
| (17 intermediate revisions by the same user not shown) | |||
| Line 12: | Line 12: | ||
** Implemented by at least XTerm, Mlterm, wezterm, and foot | ** Implemented by at least XTerm, Mlterm, wezterm, and foot | ||
** with patches out for at least VTE, alacritty, and Windows Terminal. | ** with patches out for at least VTE, alacritty, and Windows Terminal. | ||
* Kitty's protocol, advanced by Kovid Goyal. | ** also supported by iTerm2, despite having its own protocol | ||
* Kitty's protocol, advanced by Kovid Goyal, and recently (Q3 2021) supported by wezterm. | |||
* The "1337" iTerm2 protocol, also implemented by wezterm. | * The "1337" iTerm2 protocol, also implemented by wezterm. | ||
* The [https://gitlab.com/klamonte/jexer/-/wikis/jexer-images Jexer Image Sequence] from Autumn Lamonte's Jexer | * The [https://gitlab.com/klamonte/jexer/-/wikis/jexer-images Jexer Image Sequence] from Autumn Lamonte's Jexer | ||
| Line 18: | Line 19: | ||
Additionally, you can shit <i>con brio</i> all over the Linux framebuffer directly when in a framebuffer console, though I wouldn't call this a protocol. Christian Parpart (associated with [https://github.com/christianparpart/contour Contour]) is working on his "[https://github.com/christianparpart/terminal-good-image-protocol/ Terminal Good Image Protocol]", but it has not yet AFAIK seen implementation. | Additionally, you can shit <i>con brio</i> all over the Linux framebuffer directly when in a framebuffer console, though I wouldn't call this a protocol. Christian Parpart (associated with [https://github.com/christianparpart/contour Contour]) is working on his "[https://github.com/christianparpart/terminal-good-image-protocol/ Terminal Good Image Protocol]", but it has not yet AFAIK seen implementation. | ||
Having waded in the velvet seas of terminal graphics protocols for a minute now, I | Having waded in the velvet seas of terminal graphics protocols for a minute now, I was putting some of my own thoughts down as [[STEGAP]]. It ended up looking sufficiently much like Kitty's protocol that I just advocate terminal authors use that one, though. | ||
===Sixel=== | ===Sixel=== | ||
| Line 31: | Line 32: | ||
==Transparency and z-ordering== | ==Transparency and z-ordering== | ||
These protocols differ from each other in fundamental, important ways, and forming an abstraction from them isn't trivial. Integrating them wholly into the z-ordered semantics of Notcurses was still more difficult. Notcurses works via <i>piles</i> (rendering contexts) of <i>planes</i> (drawing surfaces), totally ordered on a z axis. Higher planes blend with or obscure lower ones. When Notcurses <i>renders</i> a pile, it projects these planes down onto a single plane, solving a single surface; it <i>rasterizes</i> this virtual surface to the physical viewing area by encoding it as a set of terminal escape codes and UTF-8 data. Only cells which have changed need to be rasterized, and eliding unchanged cells is a critical optimization, usually cutting the output size down by 90% or more. Every output byte is processed several times before becoming visible (we must copy it to a kernel buffer, the terminal must read it from a kernel buffer, and the terminal must display the resulting cell), so trading some computation for fewer output bytes is well worth it. | These protocols differ from each other in fundamental, important ways, and forming an abstraction from them isn't trivial. Integrating them wholly into the z-ordered semantics of Notcurses was still more difficult. Notcurses works via <i>piles</i> (rendering contexts) of <i>planes</i> (drawing surfaces), with each pile's planes totally ordered on a z axis. Higher planes blend with or obscure lower ones. When Notcurses <i>renders</i> a pile, it projects these planes down onto a single plane, solving a single surface; it <i>rasterizes</i> this virtual surface to the physical viewing area by encoding it as a set of terminal escape codes and UTF-8 data. Only cells which have changed need to be rasterized, and eliding unchanged cells is a critical optimization, usually cutting the output size down by 90% or more. Every output byte is processed several times before becoming visible (we must copy it to a kernel buffer, the terminal must read it from a kernel buffer, and the terminal must display the resulting cell), so trading some computation for fewer output bytes is well worth it. | ||
The elegant model of planes of fixed-width, independent cells solving for a single cell matrix already breaks down in the presence of [[Unicode]] wide glyphs, perhaps foremost among them U+FDFD ARABIC LIGATURE BISMILLAH AR-RAHMAN AR-RAHEEM ﷽, a single glyph occupying anywhere from one to who-knows (I've seen 9) cells, depending on font, font engine, and terminal emulator. Wide characters cannot be printed in part, and printing another character anywhere atop a wide character obliterates all of the old character. Even restricting ourselves to characters two cells wide, a single character can annihilate four columns containing two characters: | The elegant model of planes of fixed-width, independent cells solving for a single cell matrix already breaks down in the presence of [[Unicode]] wide glyphs, perhaps foremost among them U+FDFD ARABIC LIGATURE BISMILLAH AR-RAHMAN AR-RAHEEM ﷽, a single glyph occupying anywhere from one to who-knows (I've seen 9) cells, depending on font, font engine, and terminal emulator. Wide characters cannot be printed in part, and printing another character anywhere atop a wide character obliterates all of the old character. Even restricting ourselves to characters two cells wide, a single character can annihilate four columns containing two characters: | ||
| Line 68: | Line 69: | ||
===Temporal vs Positional=== | ===Temporal vs Positional=== | ||
Sixel writes the provided glyph as a unit, starting at the current cursor position (assuming we've entered DECSDM aka "Sixel Display Mode", which we always do—otherwise all sixels are emitted at the upper left corner of the display). Bits not specified in the sixel are transparent (assuming the Device Control String has used the value 1 for parameter <tt>P2</tt>), and transparent pixels leave undisturbed whatever was already present at the time of emission. Writing an entirely transparent region is a no-op. Writing a new character (or new Sixel pixels) anywhere in the graphic's region will wipe out the entirety of that cell. The Sixel data thus annihilated | Sixel writes the provided glyph as a unit, starting at the current cursor position (assuming we've entered DECSDM aka "Sixel Display Mode", which we always do—otherwise all sixels are emitted at the upper left corner of the display). Bits not specified in the sixel are transparent (assuming the Device Control String has used the value 1 for parameter <tt>P2</tt>), and transparent pixels leave undisturbed whatever was already present at the time of emission. Writing an entirely transparent region is a no-op. Writing a new character (or new Sixel pixels) anywhere in the graphic's region will wipe out the entirety of that cell. The Sixel data thus annihilated are not recoverable. This means that updating the text underneath a Sixel will blow the obscuring graphics away, requiring that the affected Sixel pixels be rerendered. In the event of stacked sprixels, we would potentially need to rerender the entire stack. In a naive implementation, changing one text cell "underneath" the octopus could result in painting 781 cells (the text cell actually being changed, and the 780 cells of the octopus). This is plainly undesirable. | ||
With Kitty, a graphic is positional. All text is assumed to be at z=0. Graphics with non-negative coordinates obscure text. Graphics with negative coordinates sit underneath text. Note that this means we must choose whether a Kitty sprixel is going to have text above it, or under it, but no arbitrary single sprixel can do both. | With Kitty, a graphic is positional. All text is assumed to be at z=0. Graphics with non-negative coordinates obscure text. Graphics with negative coordinates sit underneath text. Note that this means we must choose whether a Kitty sprixel is going to have text above it, or under it, but no arbitrary single sprixel can do both. | ||
| Line 76: | Line 77: | ||
==Color quantization== | ==Color quantization== | ||
The Kitty protocol accepts arbitrary RGBA 32-bit words in the sRGB color space; our natural internal format is 24bpp RGB plus two bits of alpha, so that's great—our colorspaces match. Sixel is a palette-indexed format. Common sizes are 16-, 256-, and 1024-entry palettes (in XTerm, this is controlled with the <tt>numColorRegisters</tt> X resource. Palettes are independent unless the <tt>privateColorRegisters</tt> resource is | The Kitty protocol accepts arbitrary RGBA 32-bit words in the sRGB color space; our natural internal format is 24bpp RGB plus two bits of alpha, so that's great—our colorspaces match. Sixel is a palette-indexed format. Common sizes are 16-, 256-, and 1024-entry palettes (in XTerm, this is controlled with the <tt>numColorRegisters</tt> X resource). Palettes are independent unless the <tt>privateColorRegisters</tt> resource is explicitly set false). A color quantization step is thus necessary (note that this these quantizations can be performed in perfect parallelism when split across multiple frames/images, but not necessarily within a single frame). There are five essential color quantization algorithms: | ||
[[File:Penrosetiling.png|thumb|200px|right|These images throw libsixel for a loop; <tt>img2sixel</tt> chugs along, and then generates no output.]] | [[File:Penrosetiling.png|thumb|200px|right|These images throw libsixel for a loop; <tt>img2sixel</tt> chugs along, and then generates no output.]] | ||
| Line 84: | Line 85: | ||
* [https://en.wikipedia.org/wiki/Octree octrees], introduced by [https://sci-hub.do/10.1007/978-3-642-83492-9_20 Gervautz and Purgathofer] in 1988. | * [https://en.wikipedia.org/wiki/Octree octrees], introduced by [https://sci-hub.do/10.1007/978-3-642-83492-9_20 Gervautz and Purgathofer] in 1988. | ||
* Kohonen neural nets as introduced by [https://www.researchgate.net/profile/Anthony-Dekker/publication/232079905_Kohonen_neural_networks_for_optimal_colour_quantization/links/09e4150ba8346d8ce9000000/Kohonen-neural-networks-for-optimal-colour-quantization.pdf Dekker in NeuQuant]. Used by the [https://lib.rs/crates/color_quant color_quant crate]. | * Kohonen neural nets as introduced by [https://www.researchgate.net/profile/Anthony-Dekker/publication/232079905_Kohonen_neural_networks_for_optimal_colour_quantization/links/09e4150ba8346d8ce9000000/Kohonen-neural-networks-for-optimal-colour-quantization.pdf Dekker in NeuQuant]. Used by the [https://lib.rs/crates/color_quant color_quant crate]. | ||
** You'll also see [https://ieeexplore.ieee.org/document/939492 Hopfield nets] | |||
* K-means clustering, perhaps best known from [https://exoticorn.github.io/exoquant-rs/exoquant/ exoquant]. | * K-means clustering, perhaps best known from [https://exoticorn.github.io/exoquant-rs/exoquant/ exoquant]. | ||
| Line 121: | Line 123: | ||
===A False Path: Font reprogramming=== | ===A False Path: Font reprogramming=== | ||
"But dank," you ask, "what if we were to seize the means of glyph production, aka the font?" Not only is that a non-portable unholy fucking mess, you still only get two colors in a stream-written glyph. Colors matter as much if not more than resolution. The NES and the SNES had basically the same resolution; the main reason why Super Mario World looked so much better than Super Mario Bros. was because it had a much wider palette. | "But dank," you ask, "what if we were to seize the means of glyph production, aka the font?" Not only is that a non-portable unholy fucking mess, you still only get two colors in a stream-written glyph. Colors matter as much if not more than resolution. The NES and the SNES had basically the same resolution; the main reason why Super Mario World looked so much better than Super Mario Bros. was because it had a much wider palette (and more memory). | ||
==Implementation details== | ==Implementation details== | ||
The most complex expression of Sixel requires four phases: clear the old bitmaps (possibly overlapping with new bitmaps), draw any updated text that's visible through transparent areas of the bitmaps, draw the bitmaps, and draw any text (updated or not) that's visible above. If cuts are employed, this can be reduced to three phases, since the text above the bitmap can be printed below the bitmap. Kitty never requires more than two phases (update text, update bitmaps), since ordering isn't relevant in the Kitty protocol. | The most complex expression of Sixel requires four phases: clear the old bitmaps (possibly overlapping with new bitmaps), draw any updated text that's visible through transparent areas of the bitmaps, draw the bitmaps, and draw any text (updated or not) that's visible above. If cuts are employed, this can be reduced to three phases, since the text above the bitmap can be printed below the bitmap. Kitty never requires more than two phases (update text, update bitmaps), since ordering isn't relevant in the Kitty protocol. | ||
It's generally necessary to immediately follow a Kitty deletion command with the redraw, if it's being replaced. The animation extension coming in 0.20.0 ought improve significantly on this situation. Clearing a Sixel requires an overwrite with text or non-transparent sixels, or a screen clear/rectangular delete. In the case where a Sixel to be destroyed and a Sixel to be painted overlap, clearing the first sixel might require damaging cells underneath the second, hence the strong dependencies between the first three phases of Sixel rendering. | It's generally necessary to immediately follow a Kitty deletion command with the redraw, if it's being replaced. The animation extension coming in 0.20.0 ought improve significantly on this situation (note: it did). Clearing a Sixel requires an overwrite with text or non-transparent sixels, or a screen clear/rectangular delete. In the case where a Sixel to be destroyed and a Sixel to be painted overlap, clearing the first sixel might require damaging cells underneath the second, hence the strong dependencies between the first three phases of Sixel rendering. | ||
Some terminals can draw Sixels more quickly when the <tt>P2</tt> parameter is 0 than when it is 1 (foot explicitly makes this claim). We need 1 if we have transparent pixels, but we can use 0 otherwise. This can be derived directly from the TAM--if and only if all TAM cells are 0, we can set P2=0. | Some terminals can draw Sixels more quickly when the <tt>P2</tt> parameter is 0 than when it is 1 (foot explicitly makes this claim). We need 1 if we have transparent pixels, but we can use 0 otherwise. This can be derived directly from the TAM--if and only if all TAM cells are 0, we can set P2=0. | ||
==Other people's bugs== | ==Other people's bugs== | ||
* XTerm through 367 doesn't update the pixel counts returned in the <tt>TIOCGWINSZ</tt> <tt>ioctl(2)</tt> when the font changes. I submitted a bug report for this, and it was [https://invisible-island.net/xterm/xterm.log.html fixed in Patch #368]. | * XTerm through 367 doesn't update the pixel counts returned in the <tt>TIOCGWINSZ</tt> <tt>ioctl(2)</tt> when the font changes. I submitted a bug report for this, and it was [https://invisible-island.net/xterm/xterm.log.html#xterm_368 fixed in Patch #368]. | ||
* XTerm through at least 366 progressively slows down as Sixels are written in alternate mode, even if they are cleared. I submitted a bug report for this. | * XTerm through at least 366 progressively slows down as Sixels are written in alternate mode, even if they are cleared. I submitted a bug report for this. | ||
* Mlterm through at least 3.9.0 makes the cursor visible after emitting a Sixel, even if it was hidden before. | * XTerm through 369 doesn't clear sixels with <tt>DECERA</tt>, <tt>DECFRA</tt>, etc. I [https://groups.google.com/g/notcurses/c/3ZBjA_v0EvQ submitted a patch for this], which was merged for [https://invisible-island.net/xterm/xterm.log.html#xterm_370 XTerm 370]. | ||
* Mlterm through at least 3.9.0 makes the cursor visible after emitting a Sixel, even if it was hidden before. I [https://github.com/arakiken/mlterm/issues/24 submitted a bug] for this. | |||
* MLterm through 3.9.1 inverts the broadly-interpreted sense of DECSDM. I [https://github.com/arakiken/mlterm/pull/23 submitted a PR] for this, which was merged. | * MLterm through 3.9.1 inverts the broadly-interpreted sense of DECSDM. I [https://github.com/arakiken/mlterm/pull/23 submitted a PR] for this, which was merged. | ||
* Kitty through 0.19.3 scrolls the screen when an emitted image touches the [https://github.com/kovidgoyal/kitty/commit/83bbcf0aa156b42eacd8503094d9d50a5638e898 lowest line]. I submitted a bug for this; the <tt>C=1</tt> modifier, available in 0.20.0, inhibits this scrolling. | * Kitty through 0.19.3 scrolls the screen when an emitted image touches the [https://github.com/kovidgoyal/kitty/commit/83bbcf0aa156b42eacd8503094d9d50a5638e898 lowest line]. I submitted a bug for this; the <tt>C=1</tt> modifier, available in 0.20.0, inhibits this scrolling. | ||
| Line 142: | Line 145: | ||
* The [https://iterm2.com/documentation-images.html iTerm2 graphics protocol] | * The [https://iterm2.com/documentation-images.html iTerm2 graphics protocol] | ||
* Thomas Dickey's [https://invisible-island.net/xterm/ctlseqs/ctlseqs.html XTerm Control Sequences] document | * Thomas Dickey's [https://invisible-island.net/xterm/ctlseqs/ctlseqs.html XTerm Control Sequences] document | ||
* [https://gitlab.com/klamonte/jexer/blob/master/README.md Jexer], Autumn Lamonte's Java library inspired by Borland's "Turbo" line of | * [https://gitlab.com/klamonte/jexer/blob/master/README.md Jexer], Autumn Lamonte's Java library inspired by Borland's "Turbo" line of products, is the only thing that comes close AFAIK | ||
* [https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/12 Issue #12] of the terminal-ng working group | * [https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/12 Issue #12] of the terminal-ng working group | ||
* hackerb9's [https://github.com/hackerb9/vt340test vt340test] repository | |||
'''previously: "[[Spooky_tmux_at_a_Distance|spooky tmux at a distance]]" 2021-02-20''' | '''previously: "[[Spooky_tmux_at_a_Distance|spooky tmux at a distance]]" 2021-02-20''' | ||
[[Category:Blog]] | [[Category:Blog]] | ||
[[CATEGORY: Terminals]] | |||