Heads up, I know there is formatting of commit messages needed and linting for preferences in Lua style guides.

The current code is me migrating over my working code from a private repo - not a fork of neovim - to be a pull request here. I'll work on updating the PR to be compliant, but wanted the code to be visible for comments.

In particular, I could use help in rewriting that parts of the PR that make use of Lua's io library - assuming we want to use a neovim equivalent - and to refactor parts of the code that could be improved. So looking for stronger critique, challenges, and suggestions 😄 This was an example-turned-PR, so not all of the code is high quality!

An example of doing this with the current PR:

local file = vim.img.load({
    filename = "/Users/senkwich/Pictures/org-roam-logo.png",
})

vim.img.show(file, {
    pos = { row = 8, col = 8 }, 
    backend = "iterm2",
})

@justinmk heads up, one complexity that we'll punt for now is supporting non-PNG images. I think we can write a pretty straightforward decoder for BMP & GIF, but JPEG is very complex and would /probably/ need a specialized C function to do it with the assistance of a JPEG-oriented library. This is in order to get RGB or RGBA data.

@kovidgoyal I'm assuming my understanding of pixel formats is correct in that if we fed in any other image format that was not PNG, using f=100 would not work, and we'd need to instead decode the base64 image data, figure out the format (i.e. bmp, jpeg, etc) and then extract a 24-bit RGB or a 32-bit RGBA set of data to feed in order for your protocol to work.

I don't know what iterm2's graphics protocol supports as I've only tested with png and I don't see anything mentioned on their doc page. I also don't know what sixel supports or how it works since I haven't read the documentation yet, but I imagine given the age of sixel that we'd need to support image decoding of some kind to break out rgb/rgba data.

Hm, it might not be possible to perfectly pixel align the images even with the kitty protocol without manually adding transparent borders using ImageMagic or similar

You can set the offset with 'X', but as far as I can see the right border is always aligned to the column. I wouldn't want to limit Neovim support to just that though. And indeed, your current protocol, does not seem to forbid it.

I guess that even the kitty protocol needs to rely on Neovim/ImageMagick doing some resizing, Neovide can implement it using the GPU though.

No the right border is not aligned to anything, you specify the top left cell and the image is simply rendered from there, unless you use the c and r control codes to render it in a specific area.

I do specify 'c' to give it the width I want, but it only takes an integer and seem to right align in my tests. So, the option is to resize the image before displaying, or add a border, to let Kitty resize.

A more concrete example is that I want to display an image starting from column 2.5 and end at column 37.75

@chipsenkbeil,

In that case it's probably the easiest to unify at least all of the terminal implementations to always add a transparent border and resize the image to fit to an exact cell width and height multipliers. As mentioned, Neovide could do that by itself for simple filtering, but the protocol could also be simplified to only deal with cell coordinates on the provider side. It also has the advantage that more advanced algorithms could be used.

Linear upsampling and lancos-3 downsampling, are good compromises for default resizing strategies https://entropymine.com/resamplescope/notes/browsers/. And I don't think Lancos-3 can be done in real-time on the GPU anyway, so we are not losing too much by delegating it to some library or another process.

What do you think?

Edit: Probably not to be done in this PR
Edit2: It also means that each placement potentially needs its own internal image.
Edit3: It also removes the need for cropping on the provider side. Except for protocols that don't have a placeholder style protocol and can therefore go out of floating windows, but that needs to be handled as special cases anyway.

So, if I get this right

(*) - Limited pixel-based sizing, affected by the position
Perfect Aspect Ratio = Automatically calculated with pixel accuracy when one dimension is left out
Crop support = Can show a cropped portion of the source scaled to the destination size

So, Sixel will need custom resizing and conversion anyway. And the rest would have to be limited to a common factor of only supporting grid based positioning, and grid based resizing, where one dimension can be left out to have it scale automatically.

That is an option for the core protocol though, loading custom formats, resizing and adding transparent borders for pixel based positioning could be delegated to plugins as long as Neovim provides accurate grid dimensions, and helper functions to do the calculations. And I think that's what's essentially implemented currently, if the pixel support and crop is removed from the opts table.

@justinmk @gpanders @j4james okay, I think I'm hitting a wall and need your help on this piece. I'm illustrating the problems I'm facing in two videos:

Flicker on sixel and iterm2

Below is a video of iTerm2, WezTerm, Ghostty, and Windows Terminal displaying images.

1. In the top-left, iTerm2 using the iTerm2 provider (sixel also works).
2. In the top-right, WezTerm using the sixel provider (kitty and iTerm2 also work)
3. In the bottom-left, Ghostty using the kitty provider
4. In the bottom-right, Windows Terminal using the sixel provider

The sixel and iterm2 are a bit rough because - unlike kitty - I need to manually clear the screen and redraw all images whenever something changes. This could be scrolling, could be typing over the image, could be swapping buffers within a window where an image is present, etc. I have to add a delay from when I trigger a screen redraw (via vim.cmd.mode()) to reduce the broken escape code problem below, which exacerbates the flicker.

1. nvim__redraw doesn't offer a way to specify a rectangular region of the screen to redraw. I can work to partially optimize by detecting which windows an image is over and using nvim__redraw to refresh it and only redraw those images, but otherwise seems like I'm stuck.
2. I can't tell when neovim has finished redrawing from vim.cmd.mode() (or other redraw asks), which means I have to use vim.defer_fn() with an arbitrary time (30ms) to reduce the second problem below. Is there any way to determine this from neovim? Or, @j4james, from any sort of terminal codes?
3. @j4james you'd mentioned pages to do image cropping. Is there any sort of double buffering I can take advantage of? Ideally, I could clear the screen, have neovim render itself, and then render all of the images again on top in some frame buffer and then copy it over, but my guess is nothing like that exists that will work with neovim (that already uses the alternate screen).

Broken Escape Codes

@gpanders @j4james I'm still getting parts of my sixel and iterm2 escape codes showing up (with images broken) every so often, and I think this is because I'm fighting neovim to write to the same tty, but I'm not fully sure.

I haven't noticed this problem on Windows with the Windows terminal. When I trigger a full refresh via vim.cmd.mode(), introducing a delay (of 30 milliseconds) before I write the escape codes for sixel or iterm2 seems to reduce the frequency a lot, but it can still happen. Even when using vim.api.nvim_chan_send(2, 'escape codes') that I saw with the clipboard code and OSC, I still get this problem. I changed the delay to 1 millisecond for the above video to show just how bad it can be (shows base64 encoded PNG data; sixel is similar with RGB).

Is there any way to control this more effectively? I'm currently batching all of my escape codes together to send out at once with the above nvim_chan_send. Been trying a bunch of different approaches and what I have now illustrated in the first video is the best I've got thus far.

Neovim eats terminal responses

@gpanders you may have missed my previous callout on this. Seems like CSI 14 t and CSI 16 t responses don't show up in TermResponse and I'm assuming are eaten in part of neovim's codebase. Same goes for when I try to request the current cursor position, but I didn't end up needing that code to get where I am thus far.

Without being able to get a response for one or both of those, Windows pixel sizes for individual cells can't be calculated. At least, based on what @j4james has mentioned in a recent comment.

The sixel and iterm2 are a bit rough because - unlike kitty - I need to manually clear the screen and redraw all images whenever something changes. This could be scrolling, could be typing over the image, could be swapping buffers within a window where an image is present, etc. I have to add a delay from when I trigger a screen redraw (via vim.cmd.mode()) to reduce the broken escape code problem below, which exacerbates the flicker.

If those were to be supported, the integration probably needs to be at a deeper level, in tui.c, with the data transferred there over the UI protocol. That part of the code knows exactly what has changed and needs to be redrawn. That's actually probably the case even for Kitty if you connect to a remote instance.

But it also makes the whole feature quite a bit more complex.

It might also be that it's easier to forget the absolute positioning support, and only support something similar to unicode placeholders. That way Neovim will automatically resolve all overlaps and partially visible images. And then you "just" need to refresh the cells as they are transferred to the tui.c code. But you still also need a way to transfer the actual image data there somehow.

I can't tell when neovim has finished redrawing from vim.cmd.mode() (or other redraw asks), which means I have to use vim.defer_fn() with an arbitrary time (30ms) to reduce the second problem below.

@chipsenkbeil I don't know anything about nvim scripting, so I'm just guessing here, but I noticed that nvim__redraw has a flush option. Would that not avoid the need for the deferred refresh, even if it still meant redrawing the entire screen? Because that might avoid the sixel/iterm sequences getting mixed up with other output if you could guarantee the redraw was flushed before you began the image output.

And assuming that does work, you might then be able to wrap both the call to nvim__redraw and the sixel/item images in the same synchronized output block, which should reduce the flickering.

Is there any sort of double buffering I can take advantage of? Ideally, I could clear the screen, have neovim render itself, and then render all of the images again on top in some frame buffer and then copy it over,

All of that is feasible in theory, but I know at least on Windows Terminal it isn't currently supported in the alt buffer (although I'd be happy to see if we could extend that support if there was a demand for it). There are a few other terminals that support paging, and which can potentially be used from within the alt buffer, but there are some idiosyncrasies in their implementation which makes me doubt whether they could support this use case.

The bottom line is this is probably something best left for later, if there is need for further optimization (particularly for clipping). But I think synchronized output is still the best short term solution for the flickering issue, and it's likely to be far more widely supported.

Without being able to get a response for one or both of those, Windows pixel sizes for individual cells can't be calculated. At least, based on what @j4james has mentioned in a recent comment.

Note that for Windows Terminal in particular, you could just fallback to a hardcoded cell size of 10x20 if you can't determine the cell size through any other means. That won't work on most other terminals though.

I can't tell when neovim has finished redrawing from vim.cmd.mode() (or other redraw asks), which means I have to use vim.defer_fn() with an arbitrary time (30ms) to reduce the second problem below.

@chipsenkbeil I don't know anything about nvim scripting, so I'm just guessing here, but I noticed that nvim__redraw has a flush option. Would that not avoid the need for the deferred refresh, even if it still meant redrawing the entire screen? Because that might avoid the sixel/iterm sequences getting mixed up with other output if you could guarantee the redraw was flushed before you began the image output.

And assuming that does work, you might then be able to wrap both the call to nvim__redraw and the sixel/item images in the same synchronized output block, which should reduce the flickering.

I got the synchronized output working. Reading through neovim's source, turns out it uses that mode already, but it can be disabled with an option, vim.o.termsync. So, I temporarily disable it while I refresh.

Flicker is gone, and I tested as well on nightly windows terminal and it works great for sixel!

@j4james is there any alternative I can pursue outside of CSI for windows pixel dimensions, cell or text area? If not, that'll need to be added to neovim core, I think.

@j4james is there any alternative I can pursue outside of CSI for windows pixel dimensions, cell or text area? If not, that'll need to be added to neovim core, I think.

@chipsenkbeil Not sure if you missed this comment, but I mentioned above you can just hardcode a cell size of 10x20 and it'll work on Windows Terminal. I'm assuming you already know the number of columns and rows, so if you need the window pixel dimensions you would just multiple the columns by 10 and the rows by 20 (it's a kind of virtual size, but the images are automatically scaled, so that's all you need to know).

That won't work for most other terminals, though, but it's at least a temporary solution until neovim core can support the CSI t size queries.

@j4james is there any alternative I can pursue outside of CSI for windows pixel dimensions, cell or text area? If not, that'll need to be added to neovim core, I think.

@chipsenkbeil Not sure if you missed this comment, but I mentioned above you can just hardcode a cell size of 10x20 and it'll work on Windows Terminal. I'm assuming you already know the number of columns and rows, so if you need the window pixel dimensions you would just multiple the columns by 10 and the rows by 20 (it's a kind of virtual size, but the images are automatically scaled, so that's all you need to know).

That won't work for most other terminals, though, but it's at least a temporary solution until neovim core can support the CSI t size queries.

Oh, I misread that and thought that was a fallback in case I couldn't get the size for Windows Terminal. So Windows Terminal is always 10x20? I'm checking for the presence of WT_SESSION right now to see if I'm in the Windows Terminal, which is a best guess unless you have another suggestion.

Anyway, thanks for all of your help on the sixel, escape codes, and Windows. Really sped up the development on this side :)

I'm checking for the presence of WT_SESSION right now to see if I'm in the Windows Terminal, which is a best guess unless you have another suggestion.

@chipsenkbeil I wouldn't recommend WT_SESSION - it's not guaranteed to be set, and there's nothing you should be doing that's specific to Windows Terminal anyway. The 10x20 size can potentially work on other terminals too (at the very least the Windows console, but also possibly a few commercial terminal emulators). That why I was saying to use it as a fallback, assuming nothing else worked (i.e. there is no ioctl, or ioctl is returning 0 dimensions). Because under those conditions you've got nothing to lose - hardcoding the size as 10x20 at least has a chance of working.