Re: [libmicrohttpd] ContentReaderCallback has changed behaviour in latest releases

[Christian Grothoff]

There are not really 'thread handles' on GNU systems (but -- are you on 
GNU or on W32? Either system has some limits, but they are typically so 
high that you should never hit them).

If this is a GNU/Linux system, the most likely cause is that you're out 
of memory.  You may try to set the size of the stack per thread to a 
smaller value (at your own peril if you get a stack overflow...). This 
can be done using MHD_OPTION_THREAD_STACK_SIZE.

Threads should be destroyed by MHD after use, but memory _fragmentation_ 
can still cause the OS to be unable to find a suitably large contiguous 
block of memory for a thread --- depending in particular on your 
application's memory allocation behavior.  Analyzing memory 
fragmentation issues is extremely tricky in my experience, but a major 
research funding agency told me a few years ago that writing tools to 
help developers do so wasn't research, so I cannot point you to any good 
method for diagnosing such problems.

Note that reducing the per-thread stack size is unlikely to fix memory 
fragmentation issues -- it might just take longer for them to manifest.  
Setting a smaller per-thread stack size is mostly useful on systems with 
limited memory, well-understood stack requirements and a large number of 
concurrent connections in relation to system memory.

Happy hacking!

-Christian

On 02/11/2013 12:56 PM, David J Myers wrote:
> Since switching to MHD_USE_THREAD_PER_CONNECTION, after the system has been
> running for a number of hours, I'm getting the following error message:-
> "Failed to create a thread: Resource temporarily unavailable"
> I can't tell if this a shortage of memory resources or thread handles or
> what.
> Is there something I need to do to cause threads to be destroyed after use?
> BR
> David
>
>
>
> -----Original Message-----
> From: Christian Grothoff [mailto:address@hidden
> Sent: 01 February 2013 10:55
> To: David J Myers
> Cc: 'libmicrohttpd development and user mailinglist'
> Subject: Re: ContentReaderCallback has changed behaviour in latest releases
>
> On 02/01/2013 11:32 AM, David J Myers wrote:
> > My server must stream to a number of concurrent clients. Does that not
> > therefore mean that I need a multi-threaded mode, not the
> > single-threaded external-select mode?
> External-select can handle up to 1k concurrent clients.
> > If I use the thread-per_connection mode, what is the consequence of
> > returning zero from the callback instead of blocking? Does it create a
> > new thread for the next callback?
> No, MHD will just call the callback instantly again.
> > If I use the thread-pool mode, I can't return zero from the callback,
> > but what would be the consequence of blocking in this mode? This is my
> > preferred mode.
> In thread-pool mode, blocking would cause all other connections that are
> processed by the same thread to stall.  So that's the worst kind of behavior
> as the other connections might have data ready.  Thus, the thread pool mode
> must only be used if the callbacks are "virtually non-blocking" (by
> "virtually", I mean to say that I'd not necessarily count disk IO operations
> in this).  Note that *all* modes can support concurrent clients; the only
> fundamental limitation (with respect to
> performance) of the select/poll modes is that they only use a single core.
>
>
> Happy hacking!
>
> Christian
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2013.0.2897 / Virus Database: 2639/6072 - Release Date: 01/31/13