bug#18222: 24.3.92; fork handlers in gmalloc.c can lead to deadlock

[YAMAMOTO Mitsuharu]

>>>>> On Fri, 08 Aug 2014 09:09:31 -0400, Ken Brown <kbrown@cornell.edu> said:

> malloc_enable_thread() in gmalloc.c calls pthread_atfork to set up fork 
> handlers.  There are a couple of problems with this, but the immediate 
> reason for this bug report is a problem on Cygwin that was reported in 
> the thread starting at

>    https://cygwin.com/ml/cygwin/2014-07/msg00387.html

> and continuing at

>    https://cygwin.com/ml/cygwin/2014-08/msg00001.html.

> The issue is that the 'prepare' fork handler locks the pthread_mutexes 
> prior to forking, and the ensuing processing of the fork command by the 
> Cygwin DLL leads to a call to malloc in the same thread, resulting in 
> deadlock.  This is a long-standing problem, but it was masked until 
> recently by the fact that pthread_mutexes on Cygwin were ERRORCHECK 
> mutexes by default.  As of Cygwin 1.7.31, pthread_mutexes are now NORMAL 
> mutexes by default, so the problem has shown up.

> A simple short-term workaround would be to explicitly set the mutexes to 
> be ERRORCHECK or RECURSIVE mutexes on Cygwin, thereby restoring the 
> previous behavior.  But this does not seem like the right long-term 
> solution, for the reasons explained here:

>    https://cygwin.com/ml/cygwin/2014-08/msg00161.html
>    https://cygwin.com/ml/cygwin/2014-08/msg00175.html

> I know nothing about this other than what I learned from the two 
> messages above, so I would appreciate some guidance.

Originally, gmalloc.c bundled with Emacs was not thread-safe, so I
added some mutex code as a short-term solution:

  http://lists.gnu.org/archive/html/emacs-devel/2007-06/msg01782.html

Thread-safe malloc was required mainly for GLib (via GTK+, for
example), and atfork handers were necessary because the threads
internally used by GLib were not under our control.

All the platforms I'm currently working at use their system malloc
rather than Emacs's gmalloc.c, so I don't think I can be of much help
about this issue.  If changing mutex attributes works well, then I
think that would be good enough for a workaround for upcoming 24.4
release.  For a long term solution, it might be better to think about
a transition to Cygwin's malloc (you mentioned
malloc_set_state/malloc_get_state in (*), but they are used only when
DOUG_LEA_MALLOC is defined).

*: https://cygwin.com/ml/cygwin/2014-08/msg00167.html

                                     YAMAMOTO Mitsuharu
                                mituharu@math.s.chiba-u.ac.jp