bug#31371: 26.1; Menu-bar stops working after search

[Nick Helm]

On Sun, 06 May 2018 at 22:14:11 +1200, Alan Third wrote:

> On Sun, May 06, 2018 at 02:15:57AM +1200, Nick Helm wrote:
>> 
>> Emacs menu-bar does not work after a help search on macOS.
>> 
>> Note that the Help menu does not reappear and the mouse does not move
>> correctly (slows down) while moving over the word "Help". Subsequent
>> clicks on the menu-bar do not work as expected either. 
>
> Looks like it goes into some sort of infinite loop calling
> ns_update_menubar. If you go to the left so it opens some non‐help
> menu, then go back to help it works fine (I think).
>
> The menu code seems pretty horrible to me, so this may take some
> time...

I had a look as well and I see what you mean, it's a bit of a mess.

It's a wild guess, but here's a theory: the problem happens because
mainMenu is trying to simultaneously be part NSMenu (Help's spotlight
search field and the context help topics) and part EmacsMenu (Help's
standard Lisp menu items).

When a user clicks on the menu bar, Emacs postpones the event via
menu_will_open_state, creates all the EmacsMenu menu items from Lisp and
regenerates the mouse click event with a call to
ns_check_pending_open_menu in order to actually display the menu.

The trouble is, NSMenu doesn't know anything about this. When the user
clicks it immediately displays what it thinks is the Help menu (sans the
Lisp stuff, which doesn't exist yet). When the EmacsMenu part is ready,
it regenerates the click event to display the menu, but NSMenu
interprets this as an instruction to hide the menu. This repeats for
each dragging mouse event over the menu-bar, hence we have a loop.

If this is the case, it should cause problems even by simply opening and
closing the Help menu. And I think that's what we're seeing. From
Emacs -Q, try opening and closing the Help menu (ignore search), then
click between the visible frame and the desktop a few times. After a
couple of tries, the frame cannot regain proper focus and the menu-bar
doesn't operate at all.

I had a fiddle around with a couple of ideas. The first removes the
NSMenu parts of the Help menu by creating an empty menu object and using
it to override the system's default Help. Unfortunately, this removes
the search field and the context topics, but the EmacsMenu menu items
then work as expected. I don't think NS Emacs has any Apple Help Book
files (the only entries that appear seem to be auto-generated) so this
might not be so bad. The search field is really useful for command
discovery though.

I also tried interrupting the call to ns_check_pending_open_menu in
x_activate_menubar. If you comment the call out, the menu sort of works
as expected, other than having to initially generate two events (click
twice) to get the Lisp menus to appear on each frame (one click works
for the appMenu and Help, but they only contain the NSMenu menu items,
as expected). I tried to make the call conditional on menu_state or
trackingMenu, but I haven't got it working.

Maybe I don't understand why a custom menu class is necessary, but I
think the right solution is to convert Help from EmacsMenu to NSMenu.
And, if we're going to do that, why not convert all of mainMenu and do
everything with standard NSMenu and NSMenuItem methods? 

All of the delayed events and tracking stuff seems over-complicated and
unnecessary. What am I missing?