[Gcl-devel] Re: GCL bug: o/main.c

[Camm Maguire]

Greetings!  What if we export some interface to setrlimit?  Would this
work?  I'd prefer to leave the default stack limit at maximum for
other known users of this feature if possible.

Take care,

Gene Cooperman <address@hidden> writes:

> The tarball is available at:
>     http://www.ccs.neu.edu/home/gene/tmp/BUG-glibc.tar.gz
> If you have trouble retrieving it, please tell me, and I'll mail it to you.
> 
> As for your questions below, someone replied to me about my bug report.
> Apparently, if the hard limit is RLIM_INFINITY and if we are using
> the ordinary libc.so library (not the debugging version), then when
> a child process is created, glibc will initialize the stack of
> the child process to the size specified by the parent's soft limit
> (by the parent's rl.rlim_cur).  Specifically, glibc does _not_
> initialize the child's stack to be the same size as the parent's
> stack size.
> 
> I don't know if glibc also initializes the child stack in that manner
> under other circumstances, but probably it does.
> 
> The child process is created successfully, but then when pthread_create()
> is called, it apparently creates a stack of the same size as the
> stack of the initial thread.  So, presumably, if the stack size of the
> initial thread is 100 MB, and if we create 40 threads, then we will
> hit this bug.  I haven't tried that experiment.
> 
> Based on this, I would have GCL set rl.rlim_cur to a maximum of 100 MB in GCL,
> and hope that most child processes will not want to create more than
> 10 or 20 threads.
> 
> In the case of ParGCL, I know that it will never
> create more than one new thread, and so I can get away with
> setting the maximum to 1 GB.  Less would be better, but because I didn't
> know all about GCL, I was worried about limiting possible interactions with
> the rest of GCL.
>                                                       Best wishes,
>                                                       - Gene
> 
> On Tue, Dec 20, 2005 at 10:53:52AM -0500, Camm Maguire wrote:
> > Greetings!
> > 
> > My apologies -- I assumed your first message was posted to the list --
> > now I can't find the original with the tarball.  I'd be happy to
> > reproduce what you are seeing here if you wouldn't mind resending.
> > 
> > But two questions some to mind:
> > 
> > 1) Can you break the 1Gb stack limit by creating more than one extra
> >    thread in the child?
> > 
> > 2) Is there any reason to expect that a 2Gb stack passed to setrlimit
> >    should be an error whereas a 1Gb stack should not?
> > 
> > I'm just trying to distinguish in my mind between 1) resource
> > exhaustion and 2) failure in libc error handling.
> > 
> > If 2Gb stack is alowable in principle, people can definitely make use
> > of the extra 1Gb, especially on conventional x86 Linux with shared
> > libs as the heap is limited to 1Gb.
> > 
> > Take care,
> > 
> > Gene Cooperman <address@hidden> writes:
> > 
> > > If you're confident that the o/main.c call to setrlimit() doesn't affect
> > > other potential applications of GCL, then you can just leave it as is.
> > > In ParGCL, I check the rl.rlim_cur of RLIM_STACK, and if it's too high,
> > > I reduce it to _only_ 1 GB of stack.
> > > 
> > > ParGCL now works fine in the current CVS.
> > > 
> > > In any case, please unpack the tarball, and run `make' in it, if you
> > > haven't done so yet.  It demonstrates a very simple bug.  I'm worried
> > > about other GCL applications.  For example, suppose the GCL interface
> > > to TCL/TK or some graphics package should create a child process.  
> > > Suppose,
> > > also, that the child process creates a thread.  In that case, you will
> > > definitely see the bug.  
> > > 
> > > In o/main.c, you could set the GCL rl.rlim_cur for the stack to a
> > > maximum of 1 GB.  This fixes the problem, and should not interfere
> > > with other uses within GCL.
> > >                                                   - Gene
> > > 
> > > On Sun, Dec 18, 2005 at 01:49:08PM -0500, Camm Maguire wrote:
> > > > Greetings!
> > > > 
> > > > Gene Cooperman <address@hidden> writes:
> > > > 
> > > > > Camm,
> > > > > 
> > > > > In GLC's o/main.c, GCL seems to trigger an existing bug in 
> > > > > glibc:setrlimit().
> > > > > I've filed a bug report on it through Debian, and I'm attaching my
> > > > > bug report.
> > > > > 
> > > > > In short, o/main.c attempts to use setrlimit() to set rlim_cur to 
> > > > > something
> > > > > unnaturally high (on my machine, to 0x7fb8ffff, which is about 2 GB).
> > > > > setrlimit silently accepts this, and when a child process tries to
> > > > > call pthread_create(), the child process reports:
> > > > >  pthread: Cannot allocate memory
> > > > >  pthread: pthread.c:22: main: Assertion `0 == pthread_retval' failed.
> > > > > 
> > > > > This bug affects ParGCL (and I patched ParGCL in the CVS on Friday to
> > > > > work around this), but it would affect any foreign function that
> > > > > calls pthread_create() inside a child process.
> > > > > 
> > > > 
> > > > Actually, this allowance for maximal stack usage is intentional, and
> > > > people are using it.  The stack also comes particularly into play when
> > > > using 'si::fork in our current implementation, not to mention
> > > > governing the maximal depth of gc cons marking and other deeply
> > > > recursive functions as well.  So in general we want to give the user
> > > > as much stack space as possible.  On current x86 Linux, which the heap
> > > > is ususally limited to 1Gb by shared memory placement, one actually
> > > > has access to almost twice as much in stack space!
> > > > 
> > > > What is odd is that the pthread implementation appears to be actually
> > > > copying the stack for the thread, rather than using some sort of copy
> > > > on write mechanism like fork().  Hmm, probably a good reason for this
> > > > somewhere. 
> > > > 
> > > > In any cae, I think the solution for us is to limit cssize in
> > > > configure to something reasonable when pargcl is selected.  Any iea on
> > > > what this should be?  Looks like it might be the default value divided
> > > > by the expected number of threads.  Perhaps we should document this
> > > > behavior somewhere too.
> > > > 
> > > > Take care, and thanks for your outstanding work in this area!
> > > > 
> > > > 
> > > > 
> > > > > I'm attaching my Debian bug report, and also a tarball that will let
> > > > > you easily reproduce the bug.  Just do:
> > > > >   tar zxvf BUG-glibc.tar.gz
> > > > >   cd BUG-glibc
> > > > >   make
> > > > > 
> > > > > The fix for GCL is probably to adjust the configure file.  I'll leave
> > > > > you to do that, since it could affect other issues in GCL.
> > > > > [ Tell me, if you prefer that I find and apply a fix. ]
> > > > >                                                       Best wishes,
> > > > >                                                       - Gene
> > > > > 
> > > > > 
> > > > > 
> > > > 
> > > > -- 
> > > > Camm Maguire                                            address@hidden
> > > > ==========================================================================
> > > > "The earth is but one country, and mankind its citizens."  --  
> > > > Baha'u'llah
> > > 
> > > 
> > > 
> > 
> > -- 
> > Camm Maguire                                                address@hidden
> > ==========================================================================
> > "The earth is but one country, and mankind its citizens."  --  Baha'u'llah
> 
> 
> 

-- 
Camm Maguire                                            address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah