bug#54654: libx11 libxcursor handling - Problems with big cursors not working in Java and xterm

[Danny Milosavljevic]

Hi,

After I got HiDPI displays (3840 pixels x 2160 pixels) mostly working in Guix
system, I am only left with one remaining problem:

libx11's XCreateFontCursor tries to dynamically-load libxcursor with just the
basename "libXcursor.so.1".

Because libxcursor depends on libx11, it's not possible to just add libxcursor
as a dependency TO libx11.

The problematic location is:

# libX11-1.7.3.1/src/CrGlCur.c

#define LIBXCURSOR "libXcursor.so.1"
static char libraryName[] = LIBXCURSOR;

static XModuleType
open_library (void)
{
[...]
        module =  dlopen(library, RTLD_LAZY);
[...]
}

#define GetFunc(type,name,ret) {\
[...]            _XcursorModule = open_library ();

#define CURSORFONT "cursor"             /* standard cursor fonts */

It's getting to dlopen of "LibXcursor.so.1" via XCreateFontCursor, which
calls XCreateGlyphCursor, which calls _XTryShapeCursor, which calls
open_library.

static Cursor
_XTryShapeCursor (Display           *dpy,
                  Font              source_font,
                  Font              mask_font,
                  unsigned int      source_char,
                  unsigned int      mask_char,
                  XColor _Xconst    *foreground,
                  XColor _Xconst    *background)
{
    TryShapeCursorFunc          func;

    GetFunc (TryShapeCursorFunc, "XcursorTryShapeCursor", func); <----- problem
    if (func)
        return (*func) (dpy, source_font, mask_font, source_char, mask_char,
                        foreground, background);
    return None;
}

Cursor XCreateGlyphCursor(
     register Display *dpy,
     Font source_font,
     Font mask_font,
     unsigned int source_char,
     unsigned int mask_char,
     XColor _Xconst *foreground,
     XColor _Xconst *background)
{
    Cursor cid;
    register xCreateGlyphCursorReq *req;

#ifdef USE_DYNAMIC_XCURSOR
    cid = _XTryShapeCursor (dpy, source_font, mask_font,
                            source_char, mask_char, foreground, background);
    if (cid)
        return cid;
#endif
    LockDisplay(dpy);
    GetReq(CreateGlyphCursor, req);
    cid = req->cid = XAllocID(dpy);
    req->source = source_font;
    req->mask = mask_font;
    req->sourceChar = source_char;
    req->maskChar = mask_char;
    req->foreRed = foreground->red;
    req->foreGreen = foreground->green;
    req->foreBlue = foreground->blue;
    req->backRed = background->red;
    req->backGreen = background->green;
    req->backBlue = background->blue;
    UnlockDisplay(dpy);
    SyncHandle();
    return (cid);
}

Cursor XCreateFontCursor(
        Display *dpy,
        unsigned int which)
{
        /*
         * the cursor font contains the shape glyph followed by the mask
         * glyph; so character position 0 contains a shape, 1 the mask for 0,
         * 2 a shape, etc.  <X11/cursorfont.h> contains hash define names
         * for all of these.
         */

        if (dpy->cursor_font == None) {
            dpy->cursor_font = XLoadFont (dpy, CURSORFONT);
            if (dpy->cursor_font == None) return None;
        }

        return XCreateGlyphCursor (dpy, dpy->cursor_font, dpy->cursor_font,
                                   which, which + 1, &foreground, &background);
}

Cursor XCreateGlyphCursor(
     register Display *dpy,
     Font source_font,
     Font mask_font,
     unsigned int source_char,
     unsigned int mask_char,
     XColor _Xconst *foreground,
     XColor _Xconst *background)
{
    Cursor cid;
    register xCreateGlyphCursorReq *req;

#ifdef USE_DYNAMIC_XCURSOR
    cid = _XTryShapeCursor (dpy, source_font, mask_font,
                            source_char, mask_char, foreground, background);
    if (cid)
        return cid;
#endif
    LockDisplay(dpy);
    GetReq(CreateGlyphCursor, req);
    cid = req->cid = XAllocID(dpy);
    req->source = source_font;
    req->mask = mask_font;
    req->sourceChar = source_char;
    req->maskChar = mask_char;
    req->foreRed = foreground->red;
    req->foreGreen = foreground->green;
    req->foreBlue = foreground->blue;
    req->backRed = background->red;
    req->backGreen = background->green;
    req->backBlue = background->blue;
    UnlockDisplay(dpy);
    SyncHandle();
    return (cid);
}

To test, to enable HiDPI:

1. Put big mouse cursors into .icons/default/cursors
2. Invoke: gsettings set org.gnome.desktop.interface cursor-size 64
3. Add to .xinitrc: xsetroot -cursor_name left_ptr # does it again
4. xrdb merge: Xft.dpi: 163

To test whether it's actually because of that (don't do it for too long):

1. Find the /gnu/store location for the libx11 used by openjdk.
2. cp -r /gnu/store/whatever-libx11* /tmp/foo
3. mount -o bind /tmp/foo /gnu/store/whatever-libx11
4. cp `guix build libxcursor`/lib/libXcursor.so* /tmp/foo/lib
5. Call your fancy Java program (for example IntelliJ IDEA)

Not sure what the best way forward is.

Possible alternative fixes:

a. Patch openjdk and xterm (and who knows what) such that it also does the
   XcursorTryShapeCursor before it calls XCreateFontCursor.
   That's a lot of work.
   Note: There are no libxcursor or XLoadFont bindings in openjdk yet,
   and one is not supposed to access Display->cursor_font from outside.
   Note: xterm use libxcursor--but for something else. So it still
   doesn't work (because it doesn't call XcursorTryShapeCursor).

b. Patch guix such that the libx11 package used by libxcursor is one without
   reference to libxcursor, but a new user-visible libx11 package actually
   would just link libxcursor in. Not sure whether that's such a good
   idea--it could increase the size of everything and have two libx11
   libraries loaded in the same user process, no?

c. Inline the build of libxcursor into the build of libx11.
   If we did that, we would add libxrender libxfixes to libx11's inputs.
!  I think this would actually be nice, especially since libxcursor seems
   to rarely change anyway (last change was in 2015).
   But even then, libxcursor depends on libxrender and libxfixes, both of
   which depends on libx11.

Non-solution steps (that's asking too much from the user--shared libraries
are an implementation detail):

1. Add an environment variable XCURSOR_RUNPATH that the user has to set
2. Patch our libx11 package to read the environment variable in GetFunc
   instead of using the global variable.
3. Add search-path or something to libxcursor.

I-don't-know-how-to-do-that-probably-bad-solution steps:

1. Add an environment variable XCURSOR_RUNPATH (see attachment)
2. Patch our libx11 package to read the environment variable in GetFunc
   instead of using the global variable. (add such a variable to
   search-path or something)
3. Patch our libxcursor package to set search-path
4. Patch our libx11 package to propagate ,(delay libxcursor)

All of those have their pros and cons.

Unfortunately, any fix causes a massive rebuild of guix, except "a".

What to do?

P.S. Same problem exists in nix.

libx11-xcursor-use-env-var.patch
Description: Text Data

pgpt6aughcJ1i.pgp
Description: OpenPGP digital signature