bug#38024: 27.0.50; icomplete sometimes fails to show completions after backward-killing words

[Kévin Le Gouguec]

Hello,

tl;dr unless I'm mistaken, sometimes `while-no-input' returns t despite
no input having arrived; this prevents `icomplete-exhibit' from
displaying completion candidates.

In more details:

This is a bit of a heisenbug; hopefully someone out there will know what
knobs to tweak to find more information about this.

>From emacs -Q:

    M-x icomplete-mode RET
    ;; Find a way to set `default-directory' to some long-ish path,
    ;; e.g. with:
    M-x find-library RET icomplete RET
    C-x C-f
    ;; icomplete should show a bunch of completions.
    M-DEL
    M-DEL
    …

After each `backward-kill-word', icomplete usually displays new
completion candidates with no further user input; sometimes however, no
candidates show up until the user does something (e.g. press TAB or
start typing).

Commit 5860fd3 (Make icomplete-exhibit actually work when navigating up
directories) tried to fix this, but AFAICT it merely reduces the odds of
the bug happening: I can still reproduce it, although less often.  Also,
this new call to `redisplay' causes some noticeable flickering when
typing characters: the completions blink in and out as I fill in the
prompt.

I've tinkered with icomplete.el (commenting out the call to `redisplay'
since it makes the bug harder to trigger); the closest I got to
pin-pointing the cause for this issue was while goofing around in
`icomplete-exhibit', specifically with the `while-no-input' part:

          (let* (…
                 (text (while-no-input
                         (icomplete-completions
                          …)))
                 …)
            ;; Do nothing if while-no-input was aborted.
            (when (stringp text)
              update overlay…))

I replaced (when (stringp text) …) with (if (stringp text) (progn …)
(message "text is %s" text)): it seems that text is t when icomplete
fails to show updated candidates, which according to `while-no-input'
means that some input arrived…

However, when I tried to debug `while-no-input', I got stumped: I tried
to do some more printf-debugging[1], but all I observe is that
sometimes, the code seems to be interrupted between

         (let ((throw-on-input ',catch-sym)
               val)
and
           (setq val (or (input-pending-p)
                         (progn ,@body)))

I tried to find where exactly the code gets interrupted, but I could not
get a consistent answer; sometimes it seems to be before executing body,
sometimes it seems to be somewhere inside `icomplete-completions'…

What would be the best way to investigate this further?

Thank you for your time.


[1] With this patch:

icomplete-debug.patch
Description: Text Data

When the bug happens, I get the following messages:

    after with-local-quit
    after catch and let
    text is t


PS: It just occured to me that I should search debbugs for
    "while-no-input" in addition to "icomplete".  I just did that;
    bug#15042 looks like it could maybe be related?


In GNU Emacs 27.0.50 (build 6, i686-pc-linux-gnu, GTK+ Version 3.24.5, cairo 
version 1.16.0)
 of 2019-10-21 built on little-buster
Repository revision: 61fb5214816ef3d57e676d900e499ffcd079a1f9
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12004000
System Description: Debian GNU/Linux 10 (buster)

Configured using:
 'configure --with-xwidgets --with-cairo'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS XWIDGETS
LIBSYSTEMD JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix