Spriteful TErminal GrAphics Protocol: Difference between revisions
No edit summary |
|||
| (15 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
STEGAP is my proposal for a terminal graphics protocol facilitating bitmapped sprites combined with terminal glyphs and styling. Just being able to blit a bitmap into a terminal is of little use for libraries like [[Notcurses]]. Unfortunately, that's about all that Sixel gives you. Useful background reading might include my [[Theory and Practice of Sprixels]]. | STEGAP is my (incomplete!) proposal for a terminal graphics protocol facilitating bitmapped sprites combined with terminal glyphs and styling. Just being able to blit a bitmap into a terminal is of little use for libraries like [[Notcurses]]. Unfortunately, that's about all that Sixel gives you. Useful background reading might include my [[Theory and Practice of Sprixels]]. | ||
Previous work includes: | |||
* DEC's Sixel protocol, the most widely-implemented (and poorly-specified) of the bunch | |||
* Kovid Goyal's [https://sw.kovidgoyal.net/kitty/graphics-protocol.html Kitty graphics protocol] | |||
* Autumn Lamonte's [https://gitlab.com/klamonte/jexer/-/wikis/jexer-images Jexer image sequence] | |||
* ITerm2's [https://iterm2.com/documentation-images.html#:~:text=Inline%20Images%20Protocol-,Inline%20Images%20Protocol,8%2Dbit%2Dclean%20environment. Image Protocol] | |||
'''nota bene: this was a learning exercise. as the result ended up looking a good deal like Kovid's protocol (see above), i encourage terminal developers to implement that protocol, as it is a major advance over Sixel. if you want to extend it with things discussed below, hit me up.''' | |||
==Goals as a toolkit developer== | ==Goals as a toolkit developer== | ||
| Line 7: | Line 15: | ||
* provide a given bitmap in as few bytes as possible | * provide a given bitmap in as few bytes as possible | ||
* associate a bitmap with an identifier. this bitmap might not be wholly opaque—transparent pixels are of critical importance, translucency less so. i ought be able to reload the bitmap (keeping the size constant), and have it redrawn without flicker. | * associate a bitmap with an identifier. this bitmap might not be wholly opaque—transparent pixels are of critical importance, translucency less so. i ought be able to reload the bitmap (keeping the size constant), and have it redrawn without flicker. | ||
** i ought be able to have an identifier generated for me if i so desire | |||
* draw text atop the bitmap without a background color, so that the graphic is not obscured except where the glyph is defined | * draw text atop the bitmap without a background color, so that the graphic is not obscured except where the glyph is defined | ||
* move the bitmap in a flicker-free way elsewhere in the visible area | * move the bitmap in a flicker-free way elsewhere in the visible area | ||
| Line 31: | Line 40: | ||
* ESC (0x1b, 27) starts a new control sequence, terminating any ongoing one. This is necessary to conform to widespread existing behavior, but it is unfortunate, as it means we can't blithely write arbitrary bytes. | * ESC (0x1b, 27) starts a new control sequence, terminating any ongoing one. This is necessary to conform to widespread existing behavior, but it is unfortunate, as it means we can't blithely write arbitrary bytes. | ||
** If we can discard this restriction, we can eliminate the unwieldy escaping in [[#Minimally encoded BGRA|BGRA]], and the biasing in [[#Unencoded greyscale|greyscale]]. | ** If we can discard this restriction, we can eliminate the unwieldy escaping in [[#Minimally encoded BGRA|BGRA]], and the biasing in [[#Unencoded greyscale|greyscale]]. | ||
** Evaluate suggestions against the [https://vt100.net/emu/dec_ansi_parser Williams state machine] | |||
* We mustn't tread on any defined [https://invisible-island.net/xterm/ctlseqs/ctlseqs.html XTerm control sequences]. | * We mustn't tread on any defined [https://invisible-island.net/xterm/ctlseqs/ctlseqs.html XTerm control sequences]. | ||
| Line 39: | Line 49: | ||
Want to indicate: | Want to indicate: | ||
* Command (load, delete, move) | * Command (load, reload, delete, move) | ||
* Identifier | * Identifier | ||
* Whether our graphic is entirely opaque (ala Sixel's <tt>P2=0</tt>); this can facilitate powerful optimizations in some terminals | * Whether our graphic is entirely opaque (ala Sixel's <tt>P2=0</tt>); this can facilitate powerful optimizations in some terminals | ||
| Line 48: | Line 58: | ||
'''FIXME extensions?''' | '''FIXME extensions?''' | ||
===Loads=== | |||
Allocates a graphic identifier, and loads it with data. The graphic will be drawn subject to [[#Graphic placement|placement rules]], unless it would be entirely off-screen, in which case it is not drawn (but still loaded, and possibly brought on-screen with a [[#Moves|move]]). A graphic which is entirely off-screen is never entered into the [[#Scrollback region|scrollback region]], and needn't (but may) be preserved when entering or leaving the [[#Alternate screen|alternate screen]]. If the identifier is already in use, no action is taken. If the identifier is -1, the terminal ought return an unused identifier. '''FIXME what if all possible identifiers are in use?''' | |||
<i>Parameters: identifier, opacity, geometry, scrolling, origin, data format, bulk data</i> | |||
===Reloads=== | |||
Reloads the specified graphic with new data. Any part of the graphic which is visible <b>must</b> be updated; this update <b>should</b> be performed without flicker. The new data must be the same geometry as the old data, but it needn't be the same format. The entirety of the data must be provided. '''FIXME we'll want some kind of animation-friendly partial reload.''' If the specified graphic does not exist, no action is taken. If the provided data is a container, and the container yields geometry different from that associated with the existing graphic, no action is taken. | |||
<i>Parameters: identifier, data format, bulk data</i> | |||
===Deletes=== | ===Deletes=== | ||
Erases the specified graphic, making visible any content which it had obscured. | Erases the specified graphic, making visible any content which it had obscured. If the identifier does not exist, no action is taken. If the identifier is -1, all graphics are deleted, and their identifiers recycled. | ||
<i>Parameters: identifier</i> | <i>Parameters: identifier</i> | ||
| Line 106: | Line 126: | ||
Pixels above the top of the visible area must be ignored. | Pixels above the top of the visible area must be ignored. | ||
Pixels beyond the bottom of the visible area depend on whether [[#Scrolling]] is in play. | Pixels beyond the bottom of the visible area depend on whether [[#Scrolling|scrolling]] is in play. | ||
==Graphic placement== | ==Graphic placement== | ||
| Line 112: | Line 132: | ||
* LTR: graphics are placed with their topmost, leftmost pixel at the current location of the cursor, starting in the top left pixel of the cell. | * LTR: graphics are placed with their topmost, leftmost pixel at the current location of the cursor, starting in the top left pixel of the cell. | ||
* RTL: graphics are placed with their topmost, rightmost pixel at the current location of the cursor, starting in the top right pixel of the cell. | * RTL: graphics are placed with their topmost, rightmost pixel at the current location of the cursor, starting in the top right pixel of the cell. | ||
* If text is being drawn top-to-bottom, you can fuck off. | * If text is being drawn top-to-bottom, you can fuck off for now (and probably already are). | ||
Two arguments allow for pixel offsets to be applied to this implicit origin; the offsets may be negative. Note that this might lead to a negative horizontal and/or vertical coordinate for the graphic's origin; see [[#Beyond the visible area|above]] to deal with this situation. | Two arguments allow for pixel offsets to be applied to this implicit origin; the offsets may be negative. Note that this might lead to a negative horizontal and/or vertical coordinate for the graphic's origin; see [[#Beyond the visible area|above]] to deal with this situation. | ||
| Line 129: | Line 149: | ||
Pixels drawn in the lowermost, rightmost cell of the visible area <b>must not</b>, in and of themselves, result in scrolling. | Pixels drawn in the lowermost, rightmost cell of the visible area <b>must not</b>, in and of themselves, result in scrolling. | ||
Pixels logically below the visible area must be ignored if scrolling is disabled. If scrolling is enabled (see [[#Scrolling|below]]), such pixels must cause scrolling equivalent to the ceiling of (rows beyond the bottom / cell height), both measured in pixels. For instance, if graphics scrolling is enabled, and cells are 10 pixels tall, and the cursor is on the bottom row when the graphic is emitted, and the graphic is 32 pixels tall, the terminal ought scroll up three rows (10 can be displayed, 32 - 10 = 22, ⌈22 / 10⌉ = 3). If scrolling is disabled, the terminal < | Pixels logically below the visible area must be ignored if scrolling is disabled. If scrolling is enabled (see [[#Scrolling|below]]), such pixels must cause scrolling equivalent to the ceiling of (rows beyond the bottom / cell height), both measured in pixels. For instance, if graphics scrolling is enabled, and cells are 10 pixels tall, and the cursor is on the bottom row when the graphic is emitted, and the graphic is 32 pixels tall, the terminal ought scroll up three rows (10 can be displayed, 32 - 10 = 22, ⌈22 / 10⌉ = 3). | ||
If scrolling is disabled, the terminal <b>must not</b> make such pixels visible if the visible area is made taller. Rationale: this could result in graphics being drawn over prompts, to the confusion of the user. | |||
==Terminal obligations== | ==Terminal obligations== | ||
| Line 146: | Line 168: | ||
'''FIXME operating parameters?''' | '''FIXME operating parameters?''' | ||
===Coexistence with other graphics protocols=== | ===Coexistence with text and other graphics protocols=== | ||
'''FIXME''' | '''FIXME''' | ||
| Line 156: | Line 178: | ||
===Scrollback buffer=== | ===Scrollback buffer=== | ||
If a line of text is available for scrollback, any graphics present on that line <b>should</b> be stored in common, and faithfully reproduced during a scrollback, unless they are deleted or moved. The identifiers of any such graphics remain reserved until the graphics are erased, or no longer relevant to the scrollback buffer, unless the terminal doesn't support graphics in the scrollback buffer, in which case their identifiers <b>must</b> be recycled. If such an offscreen graphic is reloaded, the reload <b>should</b> be faithfully reproduced. | If a line of text is available for scrollback, any graphics present on that line <b>should</b> be stored in common, and faithfully reproduced during a scrollback, unless they are deleted or moved. The identifiers of any such graphics remain reserved until the graphics are erased, or no longer relevant to the scrollback buffer, unless the terminal doesn't support graphics in the scrollback buffer, in which case their identifiers <b>must</b> be recycled. If such an offscreen graphic is reloaded, the reload <b>should</b> be faithfully reproduced. | ||
If scrollback is preserved to disk, graphics in the preserved region <b>should</b> be likewise preserved. If graphics are preserved, their identifiers <b>should</b> be preserved, and made available to new contexts. The means of referencing them from within the preserved content is entirely up to the terminal. | |||
A sufficiently negative placement offset can move or even construct a graphic logically in the scrollback region. Such a construction <b>should not</b> be faithfully reproduced. Instead, such a graphic <b>should</b> be considered "wholly off-screen", and subject to the rules described above in [[#Loads|Loads]]. | |||
===Alternate screen=== | ===Alternate screen=== | ||
| Line 164: | Line 190: | ||
* There is no native expression of palette-indexed color, nor HSV (Sixel) | * There is no native expression of palette-indexed color, nor HSV (Sixel) | ||
* There is no filesystem-based data exchange (Kitty, iTerm) | * There is no filesystem-based data exchange (Kitty, iTerm) | ||
* | * There is no terminal-side scaling (Kitty) | ||
[[CATEGORY: Terminals]] | |||
[[CATEGORY: Projects]] | |||