Re: Rethinking the design of xwidgets

[joakim]

Akira Kyle <ak@akirakyle.com> writes:

> Hi all,
>
> If you don't care about the motivation and background for this problem
> skip the next two paragraphs.
>
> One of my primary uses of emacs is as a scientific notebook utilizing
> org mode's support for literate programming to interweave prose,
> LaTeX, and code. I find this to be far more powerful and comfortable
> than using the Jupyter notebooks which have quickly become incredibly
> popular for such scientific computing. However the biggest pain point
> I have is due to Emacs' relatively poor capabilities for interfacing
> with objects other than text. Sure Emacs can display images ok, but
> I'm not aware of any way to get emacs to display more dynamic
> content. Furthermore I always want to be able to do more inside Emacs,
> and the one other application I spend as much time in as I spend in
> Emacs is a browser. Unfortunately I can't even really use eww for the
> simplest of my web browsing tasks such as looking up documentation as
> it usually fails to adequately render the variety of ways websites
> embed mathematics.
>
> This has all led me to be interested in ways to enhance Emacs' display
> capabilities. The two projects I've come across that do this are the
> experimental xwidgets configuration and the Emacs Application
> Framework (EAF) [1]. Unfortunately EAF uses Qt to reparent a window
> into emacs and Python to do two way RPC with elisp. IMO it isn't a
> very emacsy way to enhance Emacs' display capabilites and is limited
> to just taking over a whole window rather than being embedded as part
> of a buffer (key to supporting dynamic content such as an interactive
> inline plot inside an org mode document). That leaves xwidgets which
> seems to be somewhat incomplete, unused, and neglected. I've been
> poking around the xwidgets code and I really like the concept, however
> the implementation seems to need some more thought and work.
>
> I see two fundamental issues with the current xwidget code. The first
> is that in order to add a new type of widget, one must integrate it
> into the C portion of Emacs' codebase and expose relevant
> functionality to elisp. The difficulty of navigating how xwidgets are
> integrated into redisplay and how to expose the inherently event
> driven nature of GUI toolkits into elisp in a safe way is a big hurdle
> to hacking on xwidgets. In the last decade of this features existence
> no one has added any additional xwidgets besides the original webkit
> one (see make-xwidget). This despite the fact that xwidgets were
> originally designed to allow embedding of any gtk widgets such as
> buttons, sliders, image views, opengl contexts [2]. Recently support
> for webkit using the native cocoa framework was added for the ns
> toolkit demonstrating that this feature need not be tied to a specific
> GUI toolkit.  The second issue is that most interesting applications
> of xwidgets will likely need to use some external C
> library. Historically, package maintainers have been hesitant to
> package emacs with xwidgets support due to the additional dependency
> on webkitgtk, which was often not kept current with security patches
> [3]. In the long run I don't think adding extra dependencies to emacs
> just for the sake of some specific xwidget is worth it.
>
> That leads me to my following proposal to rethink the design of
> xwidgets. Given that Emacs now has support for dynamic modules and it
> is now enabled by default, I propose exposing an interface for dynamic
> modules to define custom xwidgets. I believe this would be a good way
> to solve the two fundamental issues with the current xwidgets
> implementation that I described above. Having a well defined interface
> means one wouldn't have to deal with debugging subtle redisplay and
> lisp interpreter issues when creating a custom xwidget
> type. Furthermore any external library needed for a custom xwidget
> type will only be required if one chooses to use that xwidget type and
> would keep such dependencies out of Emacs itself.
>
> I've already done a proof of concept prototype to convince myself that
> this is possible. I first modified the xwidget display spec to include
> a field which is expected to contain a function to initialize an
> xwidget view. When redisplay acts on this spec, the xwidgets code
> calls this function with the xwidget display spec and expects a
> user_ptr Lisp_Object to be returned which wraps a pointer to a
> GtkWidget. The xwidgets code then adds this widget to an appropriate
> container that it uses to ensure the widget is moved when the position
> or clipping of its glyph changes. I then define this xwidget view
> initialization function in a dynamic module.
>
> Since every window containing the xwidget has its own view, and hence
> separate GtkWidget instance, it is then up to the dynamic module to
> define the how to handle the updating of each view based on some
> common model each is supposed to represent. This works around the
> limitation that in most GUI toolkits each widget can only have one
> view. Thus for something like the webkitgtk widget, one implementation
> might choose to do offscreen rendering (as is currently done) so that
> all windows displaying the widget are updated together when, say, one
> of them is scrolled. While another implementation might decide each
> view gets a separate view of the same webpage and scrolling one will
> not scroll the other (which is what the cocoa webkit implementation
> should probably do, but it is currently only limited to being
> displayed in a single window).
>
> I'd be interested in any thoughts anyone might have about this
> proposal and if some changes along this line might have some chance of
> being accepted in master someday. I've always wanted to dig into the
> nity grity c code of emacs and this has finally given me the excuse to
> do so! Akira

As the original author of the xwidgets code, I'm happy that you are
taking an interest in moving the idea forward, and I like your ideas.

I had these ideas in the same vein originally:

- Use the gnome object system to create and manipulate xwidgets
dynamically. There was a prototype for this in the original xwidget
branch. It worked, but it was tedious figuring out how to map between
elisp types and gtk types.

- In those cases when the gobject bridge didnt behave, use the emacs
  module system for shims(the approach you describe above)

Some questions:
- The original xwidgets were designed to allow several views to the same
xwidget using gtk reflection, as you describe. When I tried using
separate gtk controllers on screen, they didnt sync very well. Did you
try this and did it work well?

When I worked on this I tried several different gtk components such as
sliders etc. Some of the worked flaky, some worked rather well. What is
you current experience trying different components?

Anyway, thanks again for having a look at this!


>
> [1] https://github.com/manateelazycat/emacs-application-framework
> [2] https://github.com/jave/xwidget-emacs
> [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843462
>
-- 
Joakim Verona
joakim@verona.se