[GNUnet-SVN] r19508 - in Extractor: . doc src/main

[gnunet]

Author: grothoff
Date: 2012-01-29 17:28:19 +0100 (Sun, 29 Jan 2012)
New Revision: 19508

Modified:
   Extractor/ChangeLog
   Extractor/doc/extractor.texi
   Extractor/src/main/extractor.c
Log:
document threading issue found today (see also discussion on #gnunet

Modified: Extractor/ChangeLog
===================================================================
--- Extractor/ChangeLog 2012-01-29 15:48:23 UTC (rev 19507)
+++ Extractor/ChangeLog 2012-01-29 16:28:19 UTC (rev 19508)
@@ -1,3 +1,7 @@
+Sun Jan 29 17:27:08 CET 2012
+       Documented recently discovered issues with pthreads and 
+       out-of-process plugin executions in the manual. -CG
+
 Tue Nov 29 12:55:40 CET 2011
        Improved IPC code on W32 to use APIs correctly and make it work
        on NT 6.1. -LRN

Modified: Extractor/doc/extractor.texi
===================================================================
--- Extractor/doc/extractor.texi        2012-01-29 15:48:23 UTC (rev 19507)
+++ Extractor/doc/extractor.texi        2012-01-29 16:28:19 UTC (rev 19508)
@@ -15,7 +15,7 @@
 GNU libextractor is a GNU package.
 
 
-Copyright @copyright{} 2007, 2010 Christian Grothoff
+Copyright @copyright{} 2007, 2010, 2012 Christian Grothoff
 
 @quotation
 Permission is granted to copy, distribute and/or modify this document
@@ -535,28 +535,33 @@
 @cindex thread-safety
 @tindex enum EXTRACTOR_Options
 
-All of the functions for loading and unloading plugins, including
address@hidden|EXTRACTOR_plugin_add_defaults|} and 
@verb{|EXTRACTOR_plugin_remove_all|},
-are thread-safe and reentrant.  However, using the same plugin list
-from multiple threads at the same time is not safe.  Creating multiple
-plugin lists and using them concurrently is supported as long as
-the @code{EXTRACTOR_OPTION_IN_PROCESS} option is not used. 
+Using @gnule{} from a multi-threaded parent process requires some
+care.  The problem is that on most platforms @gnule{} starts
+sub-processes for the actual extraction work.  This is useful to
+isolate the parent process from potential bugs; however, it can cause
+problems if the parent process is multi-threaded.  The issue is that
+at the time of the fork, another thread of the application may hold a
+lock (i.e. in gettext or libc).  That lock would then never be
+released in the child process (as the other thread is not present in
+the child process).  As a result, the child process would then
+deadlock on trying to acquire the lock and never terminate.  This has
+actually been observed with a lock in GNU gettext that is triggered by
+the plugin startup code when it interacts with libltdl.
 
-Generally, @gnule{} is fully thread-safe and mostly reentrant.
+The problem can be solved by loading the plugins using the
address@hidden option, which will run @gnule{}
+in-process and thus avoid the locking issue.  In this case, all of the
+functions for loading and unloading plugins, including
address@hidden|EXTRACTOR_plugin_add_defaults|} and
address@hidden|EXTRACTOR_plugin_remove_all|}, are thread-safe and reentrant.
+However, using the same plugin list from multiple threads at the same
+time is not safe.  
+
 All plugin code is expected required to be reentrant and state-less,
 but due to the extensive use of 3rd party libraries this cannot
-be guaranteed.  Hence plugins are executed (by default) out of
-process.  This also ensures that plugins that crash do not cause
-the main application to fail as well.  
+be guaranteed.  
 
-Plugins can be executed in-process by giving the option
address@hidden when loading the plugin.  This
-option is only recommended when debugging plugins and not for
-production use.  Due to the use of shared-memory IPC the
-out-of-process execution of plugins should not be a concern for
-performance.
 
-
 @deftp {C Struct} EXTRACTOR_PluginList
 @tindex struct EXTRACTOR_PluginList
 

Modified: Extractor/src/main/extractor.c
===================================================================
--- Extractor/src/main/extractor.c      2012-01-29 15:48:23 UTC (rev 19507)
+++ Extractor/src/main/extractor.c      2012-01-29 16:28:19 UTC (rev 19508)
@@ -1096,8 +1096,7 @@
 
 
 static int
-read_all (
-    int fd,
+read_all (int fd,
          void *buf,
          size_t size)
 {
@@ -1688,6 +1687,9 @@
   fflush (plugin->cpipe_in);
   while (1)
     {
+      fprintf (stderr, "Reading header from PID %u (plugin %s)\n",
+              plugin->cpid,
+              plugin->short_libname);
       if (0 != read_all (plugin->cpipe_out,
                         &hdr,
                         sizeof(hdr)))