Theory and Practice of Sprixels: Difference between revisions
No edit summary |
|||
| (3 intermediate revisions by the same user not shown) | |||
| Line 6: | Line 6: | ||
I've spent significant time over the past few weeks adding "sprixel" support to [[Notcurses]], culminating (thus far) in yesterday's [https://github.com/dankamongmen/notcurses/releases/tag/v2.2.4 2.2.4 release]. What's a sprixel? Numerous [https://github.com/dankamongmen/notcurses/blob/master/TERMINALS.md terminal emulators] support one or another "graphic protocol" by which bitmaps of arbitrary size can be written to the glyph-based viewing area. Showing that dynamic control of the font <b>is not</b> even approximately equivalent in power to such graphic protocols is left as an exercise for the reader (or you can scroll to the end). | I've spent significant time over the past few weeks adding "sprixel" support to [[Notcurses]], culminating (thus far) in yesterday's [https://github.com/dankamongmen/notcurses/releases/tag/v2.2.4 2.2.4 release]. What's a sprixel? Numerous [https://github.com/dankamongmen/notcurses/blob/master/TERMINALS.md terminal emulators] support one or another "graphic protocol" by which bitmaps of arbitrary size can be written to the glyph-based viewing area. Showing that dynamic control of the font <b>is not</b> even approximately equivalent in power to such graphic protocols is left as an exercise for the reader (or you can scroll to the end). | ||
Sprite + Sixel → Sprixel. | |||
==Sprixel flavors== | ==Sprixel flavors== | ||
| Line 77: | Line 79: | ||
==Color quantization== | ==Color quantization== | ||
The Kitty protocol accepts arbitrary RGBA 32-bit words in the sRGB color space; our natural internal format is 24bpp RGB plus two bits of alpha, so that's great—our colorspaces match. Sixel is a palette-indexed format. Common sizes are 16-, 256-, and 1024-entry palettes (in XTerm, this is controlled with the <tt>numColorRegisters</tt> X resource). Palettes are independent unless the <tt>privateColorRegisters</tt> resource is explicitly set false | The Kitty protocol accepts arbitrary RGBA 32-bit words in the sRGB color space; our natural internal format is 24bpp RGB plus two bits of alpha, so that's great—our colorspaces match. Sixel is a palette-indexed format. Common sizes are 16-, 256-, and 1024-entry palettes (in XTerm, this is controlled with the <tt>numColorRegisters</tt> X resource). Palettes are independent unless the <tt>privateColorRegisters</tt> resource is explicitly set false. A color quantization step is thus necessary (note that this these quantizations can be performed in perfect parallelism when split across multiple frames/images, but not necessarily within a single frame). There are five essential color quantization algorithms: | ||
[[File:Penrosetiling.png|thumb|200px|right|These images throw libsixel for a loop; <tt>img2sixel</tt> chugs along, and then generates no output.]] | [[File:Penrosetiling.png|thumb|200px|right|These images throw libsixel for a loop; <tt>img2sixel</tt> chugs along, and then generates no output.]] | ||
| Line 128: | Line 130: | ||
The most complex expression of Sixel requires four phases: clear the old bitmaps (possibly overlapping with new bitmaps), draw any updated text that's visible through transparent areas of the bitmaps, draw the bitmaps, and draw any text (updated or not) that's visible above. If cuts are employed, this can be reduced to three phases, since the text above the bitmap can be printed below the bitmap. Kitty never requires more than two phases (update text, update bitmaps), since ordering isn't relevant in the Kitty protocol. | The most complex expression of Sixel requires four phases: clear the old bitmaps (possibly overlapping with new bitmaps), draw any updated text that's visible through transparent areas of the bitmaps, draw the bitmaps, and draw any text (updated or not) that's visible above. If cuts are employed, this can be reduced to three phases, since the text above the bitmap can be printed below the bitmap. Kitty never requires more than two phases (update text, update bitmaps), since ordering isn't relevant in the Kitty protocol. | ||
It's generally necessary to immediately follow a Kitty deletion command with the redraw, if it's being replaced. The animation extension coming in 0.20.0 ought improve significantly on this situation. Clearing a Sixel requires an overwrite with text or non-transparent sixels, or a screen clear/rectangular delete. In the case where a Sixel to be destroyed and a Sixel to be painted overlap, clearing the first sixel might require damaging cells underneath the second, hence the strong dependencies between the first three phases of Sixel rendering. | It's generally necessary to immediately follow a Kitty deletion command with the redraw, if it's being replaced. The animation extension coming in 0.20.0 ought improve significantly on this situation (note: it did). Clearing a Sixel requires an overwrite with text or non-transparent sixels, or a screen clear/rectangular delete. In the case where a Sixel to be destroyed and a Sixel to be painted overlap, clearing the first sixel might require damaging cells underneath the second, hence the strong dependencies between the first three phases of Sixel rendering. | ||
Some terminals can draw Sixels more quickly when the <tt>P2</tt> parameter is 0 than when it is 1 (foot explicitly makes this claim). We need 1 if we have transparent pixels, but we can use 0 otherwise. This can be derived directly from the TAM--if and only if all TAM cells are 0, we can set P2=0. | Some terminals can draw Sixels more quickly when the <tt>P2</tt> parameter is 0 than when it is 1 (foot explicitly makes this claim). We need 1 if we have transparent pixels, but we can use 0 otherwise. This can be derived directly from the TAM--if and only if all TAM cells are 0, we can set P2=0. | ||
| Line 145: | Line 147: | ||
* The [https://iterm2.com/documentation-images.html iTerm2 graphics protocol] | * The [https://iterm2.com/documentation-images.html iTerm2 graphics protocol] | ||
* Thomas Dickey's [https://invisible-island.net/xterm/ctlseqs/ctlseqs.html XTerm Control Sequences] document | * Thomas Dickey's [https://invisible-island.net/xterm/ctlseqs/ctlseqs.html XTerm Control Sequences] document | ||
* [https://gitlab.com/klamonte/jexer/blob/master/README.md Jexer], Autumn Lamonte's Java library inspired by Borland's "Turbo" line of | * [https://gitlab.com/klamonte/jexer/blob/master/README.md Jexer], Autumn Lamonte's Java library inspired by Borland's "Turbo" line of products, is the only thing that comes close AFAIK | ||
* [https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/12 Issue #12] of the terminal-ng working group | * [https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/12 Issue #12] of the terminal-ng working group | ||
* hackerb9's [https://github.com/hackerb9/vt340test vt340test] repository | * hackerb9's [https://github.com/hackerb9/vt340test vt340test] repository | ||