Spriteful TErminal GrAphics Protocol: Difference between revisions

(6 intermediate revisions by the same user not shown)

Line 1:

STEGAP is my (incomplete!) proposal for a terminal graphics protocol facilitating bitmapped sprites combined with terminal glyphs and styling. Just being able to blit a bitmap into a terminal is of little use for libraries like [[Notcurses]]. Unfortunately, that's about all that Sixel gives you. Useful background reading might include my [[Theory and Practice of Sprixels]].

Previous work includes:

* DEC's Sixel protocol, the most widely-implemented (and poorly-specified) of the bunch

* Kovid Goyal's [https://sw.kovidgoyal.net/kitty/graphics-protocol.html Kitty graphics protocol]

* Autumn Lamonte's [https://gitlab.com/klamonte/jexer/-/wikis/jexer-images Jexer image sequence]

* ITerm2's [https://iterm2.com/documentation-images.html#:~:text=Inline%20Images%20Protocol-,Inline%20Images%20Protocol,8%2Dbit%2Dclean%20environment. Image Protocol]

'''nota bene: this was a learning exercise. as the result ended up looking a good deal like Kovid's protocol (see above), i encourage terminal developers to implement that protocol, as it is a major advance over Sixel. if you want to extend it with things discussed below, hit me up.'''

==Goals as a toolkit developer==

Line 32:

Line 40:

* ESC (0x1b, 27) starts a new control sequence, terminating any ongoing one. This is necessary to conform to widespread existing behavior, but it is unfortunate, as it means we can't blithely write arbitrary bytes.

** If we can discard this restriction, we can eliminate the unwieldy escaping in [[#Minimally encoded BGRA|BGRA]], and the biasing in [[#Unencoded greyscale|greyscale]].

** Evaluate suggestions against the [https://vt100.net/emu/dec_ansi_parser Williams state machine]

* We mustn't tread on any defined [https://invisible-island.net/xterm/ctlseqs/ctlseqs.html XTerm control sequences].

Line 56:

Line 65:

===Reloads===

Reloads the specified graphic with new data. Any part of the graphic which is visible must be updated; this update should be performed without flicker. The new data must be the same geometry as the old data, but it needn't be the same format. The entirety of the data must be provided. '''FIXME we'll want some kind of animation-friendly partial reload.''' If the specified graphic does not exist, no action is taken.

Reloads the specified graphic with new data. Any part of the graphic which is visible must be updated; this update should be performed without flicker. The new data must be the same geometry as the old data, but it needn't be the same format. The entirety of the data must be provided. '''FIXME we'll want some kind of animation-friendly partial reload.''' If the specified graphic does not exist, no action is taken. If the provided data is a container, and the container yields geometry different from that associated with the existing graphic, no action is taken.

Parameters: identifier, data format, bulk data

Line 140:

Line 149:

Pixels drawn in the lowermost, rightmost cell of the visible area must not, in and of themselves, result in scrolling.

Pixels logically below the visible area must be ignored if scrolling is disabled. If scrolling is enabled (see [[#Scrolling|below]]), such pixels must cause scrolling equivalent to the ceiling of (rows beyond the bottom / cell height), both measured in pixels. For instance, if graphics scrolling is enabled, and cells are 10 pixels tall, and the cursor is on the bottom row when the graphic is emitted, and the graphic is 32 pixels tall, the terminal ought scroll up three rows (10 can be displayed, 32 - 10 = 22, ⌈22 / 10⌉ = 3). If scrolling is disabled, the terminal ~~may~~ make such pixels visible if the visible area is made taller, ~~but this is not required~~.

Pixels logically below the visible area must be ignored if scrolling is disabled. If scrolling is enabled (see [[#Scrolling|below]]), such pixels must cause scrolling equivalent to the ceiling of (rows beyond the bottom / cell height), both measured in pixels. For instance, if graphics scrolling is enabled, and cells are 10 pixels tall, and the cursor is on the bottom row when the graphic is emitted, and the graphic is 32 pixels tall, the terminal ought scroll up three rows (10 can be displayed, 32 - 10 = 22, ⌈22 / 10⌉ = 3).

If scrolling is disabled, the terminal must not make such pixels visible if the visible area is made taller. Rationale: this could result in graphics being drawn over prompts, to the confusion of the user.

==Terminal obligations==

Line 180:

Line 191:

* There is no filesystem-based data exchange (Kitty, iTerm)

* There is no terminal-side scaling (Kitty)

[[CATEGORY: Terminals]]

[[CATEGORY: Projects]]

@@ Line 1: / Line 1: @@
 STEGAP is my (incomplete!) proposal for a terminal graphics protocol facilitating bitmapped sprites combined with terminal glyphs and styling. Just being able to blit a bitmap into a terminal is of little use for libraries like [[Notcurses]]. Unfortunately, that's about all that Sixel gives you. Useful background reading might include my [[Theory and Practice of Sprixels]].
+Previous work includes:
+* DEC's Sixel protocol, the most widely-implemented (and poorly-specified) of the bunch
+* Kovid Goyal's [https://sw.kovidgoyal.net/kitty/graphics-protocol.html Kitty graphics protocol]
+* Autumn Lamonte's [https://gitlab.com/klamonte/jexer/-/wikis/jexer-images Jexer image sequence]
+* ITerm2's [https://iterm2.com/documentation-images.html#:~:text=Inline%20Images%20Protocol-,Inline%20Images%20Protocol,8%2Dbit%2Dclean%20environment. Image Protocol]
+'''nota bene: this was a learning exercise. as the result ended up looking a good deal like Kovid's protocol (see above), i encourage terminal developers to implement that protocol, as it is a major advance over Sixel. if you want to extend it with things discussed below, hit me up.'''
 ==Goals as a toolkit developer==
@@ Line 32: / Line 40: @@
 * ESC (0x1b, 27) starts a new control sequence, terminating any ongoing one. This is necessary to conform to widespread existing behavior, but it is unfortunate, as it means we can't blithely write arbitrary bytes.
 ** If we can discard this restriction, we can eliminate the unwieldy escaping in [[#Minimally encoded BGRA|BGRA]], and the biasing in [[#Unencoded greyscale|greyscale]].
+** Evaluate suggestions against the [https://vt100.net/emu/dec_ansi_parser Williams state machine]
 * We mustn't tread on any defined [https://invisible-island.net/xterm/ctlseqs/ctlseqs.html XTerm control sequences].
@@ Line 56: / Line 65: @@
 ===Reloads===
-Reloads the specified graphic with new data. Any part of the graphic which is visible <b>must</b> be updated; this update <b>should</b> be performed without flicker. The new data must be the same geometry as the old data, but it needn't be the same format. The entirety of the data must be provided. '''FIXME we'll want some kind of animation-friendly partial reload.''' If the specified graphic does not exist, no action is taken.
+Reloads the specified graphic with new data. Any part of the graphic which is visible <b>must</b> be updated; this update <b>should</b> be performed without flicker. The new data must be the same geometry as the old data, but it needn't be the same format. The entirety of the data must be provided. '''FIXME we'll want some kind of animation-friendly partial reload.''' If the specified graphic does not exist, no action is taken. If the provided data is a container, and the container yields geometry different from that associated with the existing graphic, no action is taken.
 <i>Parameters: identifier, data format, bulk data</i>
@@ Line 140: / Line 149: @@
 Pixels drawn in the lowermost, rightmost cell of the visible area <b>must not</b>, in and of themselves, result in scrolling.
-Pixels logically below the visible area must be ignored if scrolling is disabled. If scrolling is enabled (see [[#Scrolling|below]]), such pixels must cause scrolling equivalent to the ceiling of (rows beyond the bottom / cell height), both measured in pixels. For instance, if graphics scrolling is enabled, and cells are 10 pixels tall, and the cursor is on the bottom row when the graphic is emitted, and the graphic is 32 pixels tall, the terminal ought scroll up three rows (10 can be displayed, 32 - 10 = 22, ⌈22 / 10⌉ = 3). If scrolling is disabled, the terminal <i>may</i> make such pixels visible if the visible area is made taller, but this is not required.
+Pixels logically below the visible area must be ignored if scrolling is disabled. If scrolling is enabled (see [[#Scrolling|below]]), such pixels must cause scrolling equivalent to the ceiling of (rows beyond the bottom / cell height), both measured in pixels. For instance, if graphics scrolling is enabled, and cells are 10 pixels tall, and the cursor is on the bottom row when the graphic is emitted, and the graphic is 32 pixels tall, the terminal ought scroll up three rows (10 can be displayed, 32 - 10 = 22, ⌈22 / 10⌉ = 3).
+If scrolling is disabled, the terminal <b>must not</b> make such pixels visible if the visible area is made taller. Rationale: this could result in graphics being drawn over prompts, to the confusion of the user.
 ==Terminal obligations==
@@ Line 180: / Line 191: @@
 * There is no filesystem-based data exchange (Kitty, iTerm)
 * There is no terminal-side scaling (Kitty)
+[[CATEGORY: Terminals]]
+[[CATEGORY: Projects]]

Spriteful TErminal GrAphics Protocol: Difference between revisions

Dank (talk | contribs)