Theory and Practice of Sprixels: Difference between revisions
No edit summary |
|||
| (4 intermediate revisions by the same user not shown) | |||
| Line 6: | Line 6: | ||
I've spent significant time over the past few weeks adding "sprixel" support to [[Notcurses]], culminating (thus far) in yesterday's [https://github.com/dankamongmen/notcurses/releases/tag/v2.2.4 2.2.4 release]. What's a sprixel? Numerous [https://github.com/dankamongmen/notcurses/blob/master/TERMINALS.md terminal emulators] support one or another "graphic protocol" by which bitmaps of arbitrary size can be written to the glyph-based viewing area. Showing that dynamic control of the font <b>is not</b> even approximately equivalent in power to such graphic protocols is left as an exercise for the reader (or you can scroll to the end). | I've spent significant time over the past few weeks adding "sprixel" support to [[Notcurses]], culminating (thus far) in yesterday's [https://github.com/dankamongmen/notcurses/releases/tag/v2.2.4 2.2.4 release]. What's a sprixel? Numerous [https://github.com/dankamongmen/notcurses/blob/master/TERMINALS.md terminal emulators] support one or another "graphic protocol" by which bitmaps of arbitrary size can be written to the glyph-based viewing area. Showing that dynamic control of the font <b>is not</b> even approximately equivalent in power to such graphic protocols is left as an exercise for the reader (or you can scroll to the end). | ||
Sprite + Sixel → Sprixel. | |||
==Sprixel flavors== | ==Sprixel flavors== | ||
| Line 69: | Line 71: | ||
===Temporal vs Positional=== | ===Temporal vs Positional=== | ||
Sixel writes the provided glyph as a unit, starting at the current cursor position (assuming we've entered DECSDM aka "Sixel Display Mode", which we always do—otherwise all sixels are emitted at the upper left corner of the display). Bits not specified in the sixel are transparent (assuming the Device Control String has used the value 1 for parameter <tt>P2</tt>), and transparent pixels leave undisturbed whatever was already present at the time of emission. Writing an entirely transparent region is a no-op. Writing a new character (or new Sixel pixels) anywhere in the graphic's region will wipe out the entirety of that cell. The Sixel data thus annihilated | Sixel writes the provided glyph as a unit, starting at the current cursor position (assuming we've entered DECSDM aka "Sixel Display Mode", which we always do—otherwise all sixels are emitted at the upper left corner of the display). Bits not specified in the sixel are transparent (assuming the Device Control String has used the value 1 for parameter <tt>P2</tt>), and transparent pixels leave undisturbed whatever was already present at the time of emission. Writing an entirely transparent region is a no-op. Writing a new character (or new Sixel pixels) anywhere in the graphic's region will wipe out the entirety of that cell. The Sixel data thus annihilated are not recoverable. This means that updating the text underneath a Sixel will blow the obscuring graphics away, requiring that the affected Sixel pixels be rerendered. In the event of stacked sprixels, we would potentially need to rerender the entire stack. In a naive implementation, changing one text cell "underneath" the octopus could result in painting 781 cells (the text cell actually being changed, and the 780 cells of the octopus). This is plainly undesirable. | ||
With Kitty, a graphic is positional. All text is assumed to be at z=0. Graphics with non-negative coordinates obscure text. Graphics with negative coordinates sit underneath text. Note that this means we must choose whether a Kitty sprixel is going to have text above it, or under it, but no arbitrary single sprixel can do both. | With Kitty, a graphic is positional. All text is assumed to be at z=0. Graphics with non-negative coordinates obscure text. Graphics with negative coordinates sit underneath text. Note that this means we must choose whether a Kitty sprixel is going to have text above it, or under it, but no arbitrary single sprixel can do both. | ||
| Line 77: | Line 79: | ||
==Color quantization== | ==Color quantization== | ||
The Kitty protocol accepts arbitrary RGBA 32-bit words in the sRGB color space; our natural internal format is 24bpp RGB plus two bits of alpha, so that's great—our colorspaces match. Sixel is a palette-indexed format. Common sizes are 16-, 256-, and 1024-entry palettes (in XTerm, this is controlled with the <tt>numColorRegisters</tt> X resource). Palettes are independent unless the <tt>privateColorRegisters</tt> resource is explicitly set false | The Kitty protocol accepts arbitrary RGBA 32-bit words in the sRGB color space; our natural internal format is 24bpp RGB plus two bits of alpha, so that's great—our colorspaces match. Sixel is a palette-indexed format. Common sizes are 16-, 256-, and 1024-entry palettes (in XTerm, this is controlled with the <tt>numColorRegisters</tt> X resource). Palettes are independent unless the <tt>privateColorRegisters</tt> resource is explicitly set false. A color quantization step is thus necessary (note that this these quantizations can be performed in perfect parallelism when split across multiple frames/images, but not necessarily within a single frame). There are five essential color quantization algorithms: | ||
[[File:Penrosetiling.png|thumb|200px|right|These images throw libsixel for a loop; <tt>img2sixel</tt> chugs along, and then generates no output.]] | [[File:Penrosetiling.png|thumb|200px|right|These images throw libsixel for a loop; <tt>img2sixel</tt> chugs along, and then generates no output.]] | ||
| Line 128: | Line 130: | ||
The most complex expression of Sixel requires four phases: clear the old bitmaps (possibly overlapping with new bitmaps), draw any updated text that's visible through transparent areas of the bitmaps, draw the bitmaps, and draw any text (updated or not) that's visible above. If cuts are employed, this can be reduced to three phases, since the text above the bitmap can be printed below the bitmap. Kitty never requires more than two phases (update text, update bitmaps), since ordering isn't relevant in the Kitty protocol. | The most complex expression of Sixel requires four phases: clear the old bitmaps (possibly overlapping with new bitmaps), draw any updated text that's visible through transparent areas of the bitmaps, draw the bitmaps, and draw any text (updated or not) that's visible above. If cuts are employed, this can be reduced to three phases, since the text above the bitmap can be printed below the bitmap. Kitty never requires more than two phases (update text, update bitmaps), since ordering isn't relevant in the Kitty protocol. | ||
It's generally necessary to immediately follow a Kitty deletion command with the redraw, if it's being replaced. The animation extension coming in 0.20.0 ought improve significantly on this situation. Clearing a Sixel requires an overwrite with text or non-transparent sixels, or a screen clear/rectangular delete. In the case where a Sixel to be destroyed and a Sixel to be painted overlap, clearing the first sixel might require damaging cells underneath the second, hence the strong dependencies between the first three phases of Sixel rendering. | It's generally necessary to immediately follow a Kitty deletion command with the redraw, if it's being replaced. The animation extension coming in 0.20.0 ought improve significantly on this situation (note: it did). Clearing a Sixel requires an overwrite with text or non-transparent sixels, or a screen clear/rectangular delete. In the case where a Sixel to be destroyed and a Sixel to be painted overlap, clearing the first sixel might require damaging cells underneath the second, hence the strong dependencies between the first three phases of Sixel rendering. | ||
Some terminals can draw Sixels more quickly when the <tt>P2</tt> parameter is 0 than when it is 1 (foot explicitly makes this claim). We need 1 if we have transparent pixels, but we can use 0 otherwise. This can be derived directly from the TAM--if and only if all TAM cells are 0, we can set P2=0. | Some terminals can draw Sixels more quickly when the <tt>P2</tt> parameter is 0 than when it is 1 (foot explicitly makes this claim). We need 1 if we have transparent pixels, but we can use 0 otherwise. This can be derived directly from the TAM--if and only if all TAM cells are 0, we can set P2=0. | ||
| Line 145: | Line 147: | ||
* The [https://iterm2.com/documentation-images.html iTerm2 graphics protocol] | * The [https://iterm2.com/documentation-images.html iTerm2 graphics protocol] | ||
* Thomas Dickey's [https://invisible-island.net/xterm/ctlseqs/ctlseqs.html XTerm Control Sequences] document | * Thomas Dickey's [https://invisible-island.net/xterm/ctlseqs/ctlseqs.html XTerm Control Sequences] document | ||
* [https://gitlab.com/klamonte/jexer/blob/master/README.md Jexer], Autumn Lamonte's Java library inspired by Borland's "Turbo" line of | * [https://gitlab.com/klamonte/jexer/blob/master/README.md Jexer], Autumn Lamonte's Java library inspired by Borland's "Turbo" line of products, is the only thing that comes close AFAIK | ||
* [https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/12 Issue #12] of the terminal-ng working group | * [https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/12 Issue #12] of the terminal-ng working group | ||
* hackerb9's [https://github.com/hackerb9/vt340test vt340test] repository | * hackerb9's [https://github.com/hackerb9/vt340test vt340test] repository | ||