Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: Akira Kyle <ak@akirakyle.com>
> Cc: emacs-devel@gnu.org
> Date: Tue, 13 Oct 2020 11:05:51 -0600
> 
> Sure, so to elaborate on my motivation: I spend a lot of my time 
> in emacs using org-mode's babel support for python, specifically 
> through emacs-jupyter [1]. This gives me much of the power of 
> Jupyter notebooks, however not all. For example when generating 
> plots using matplotlib, they are simply embedded in the buffer as 
> an image, however matplotlib also has support for interactive 
> backends [2] which allow one to zoom and pan the plot and see it 
> update in real time as each plot command is issued. It would be 
> convenient to have such interactivity directly inside the buffer 
> rather than having to use an external window, similar to the 
> interactivity offered when using matplotlib in a jupyter 
> notebook. The bokeh plotting [3] library provides even more 
> interactivity, allowing one to create complex and dynamic and 
> interactive visualizations. When used inside a jupyter notebook, 
> the plot is embedded inline, however when used with emacs-jupyter, 
> it launches an external browser to display the interactive plot 
> and it's controls. It would be fantastic if I could have such 
> interactivity inside Emacs.

I'll need to read up on those two topics, but in general, as someone
already pointed out, the Emacs display of images supports
interactivity by letting you call functions triggered by mouse
gestures on the varipous spots on the image.  If such a function
produces a zoomed-in/out image and displays it instead of the original
one, wouldn't that fulfill the need?  E.g., we already support
animated GIF images and zoom-in/out by mouse scroll, so it is
inaccurate to say that we can only display static images.

> Another use case I had in mind for my proposed improved xwidgets 
> is better pdf rendering. I use pdf-tools constantly and it's one 
> of the best pdf viewers out there (not just because it's in 
> emacs). I'll sometimes even use it to give presentations so I 
> never have to leave Emacs. However it's design inherently limits 
> its rendering performance as it must rasterize each pdf page in a 
> separate process then pipe that data to emacs to display as an 
> image.

This use case doesn't necessarily call for the same solution, but I
could think of a differently designed pdf-viewer that didn't need to
rasterize each page separately.  Again, the mouse-sensitivity feature
of our image display could perhaps be used for something like that.  I
reckon the hardest problem is to find an engine, be it a library or a
program, that could convert PDF into something Emacs can display.

> >   . their integration in the Emacs display code "needs work", 
> >   there's
> >     at least one or two places where the code which handles them 
> >     is
> >     clearly wrong -- this is semi-okay for a minor niche 
> >     feature, but
> >     not for something on which we want to build our future
> 
> Could you point me to some of those places. Maybe they are issues 
> I've already taken a look at.

E.g., the kludge in dispnew.c around line 4365.  It disables one of
the most important redisplay optimizations in Emacs, once you build
with xwidgets enabled.

> I think my pdf use case illustrates the issue with Emacs trying to 
> support every such type of content as a high level "image-like" 
> feature since you either force the application to convert 
> everything into a sub-optimal format that Emacs understands for 
> display or Emacs' must add native for such format, increasing the 
> number of external dependencies to build Emacs.
> 
> For example, consider svg support, which is my preferred image 
> format. As far is I can tell, currently Emacs rasterizes svgs 
> itself before ultimately drawing them on a cairo context.

Do we actually do that?  We use librsvg, don't we?

Or maybe you are talking about the generic layer of image handling,
which basically needs a bitmap/pixmap?

In any case, this seems to be a much more fundamental issue of how we
display images in Emacs.  Which is not necessarily the same as the
issue you raised originally.

> Since librsvg supports drawing directly on a cairo context, it would
> be much more efficient to do so, however adding support for svg
> could be easily moved out of emacs into a dynamic module and drawn
> directly onto it's own widget.

The problem with that is that handling such "widgets" in our display
engine is entirely non-trivial, as the xwidgets code clearly shows.
There are basic problems in displaying xwidgets that from my POV are
left unsolved, and I'm not sure we will be able to find satisfactory
solutions for them.  The display engine has several requirements that
aren't easy to support with widgets that want to manage themselves.

I see where you are coming from with the idea of dynamic modules, but
I'm not sure your assumption that it will make things easier for
implementing "display elements" that have their own behavior
independent of Emacs -- I'm not sure this assumption is indeed true.
The Emacs display code has a deceptively simple main design idea and
execution concept, and as long as one thinks this is all there is to
it, it could indeed seem that adding such widgets should be almost
trivial.  But the reality is that there are a lot of details to go
with the main idea, and it's the details that make implementing this
stuff more difficult than one thinks first.

I'm not trying to scare you, mind you, just point out that the ideas
you have should probably be talked through in mode detail, to see that
they are workable.  For example, did you study the various
window-redisplay optimizations, and if so, did you think how the
widgets will adapt to them?