Re: [libmicrohttpd] ContentReaderCallback has changed behaviour in latest releases

[David J Myers]

Could you explain 'external_select'? Is this a different threading model
from 'thread-per-connection'? I can't see a parameter definition for this.
Could you show me an example call to MHD_start_daemon which selects this
mode?
Regards
David


-----Original Message-----
From: Christian Grothoff [mailto:address@hidden 
Sent: 01 February 2013 09:08
To: David J Myers; libmicrohttpd development and user mailinglist
Subject: Re: ContentReaderCallback has changed behaviour in latest releases

On 02/01/2013 10:00 AM, David J Myers wrote:
> Ok, I changed the threading mode from MHD_USE_SELECT_INTERNALLY (with 
> thread
> pool) to MHD_USE_THREAD_PER_CONNECTION and this fixes the problem. I 
> guess I was getting away with it on previous releases.
Possible -- I checked and found that the error message has been there for a
long time (0.4.x or even earlier).  However, you might have had it disabled
(configure defaults changed at some point), and MHD might have been more
tolerant of busy waiting in the past.

> I'm hoping that this mode
> change will not have any unwanted effects on the behaviour of the 
> system (IP video camera).
I don't see why it would, except that you get more threads which may consume
more memory.  Note that if you use 'thread per connection', the idea is that
your code should just block instead of returning zero. If you're concerned
about performance (i.e. RAM) on an embedded system, 'external select' is
usually the best choice IMO. Thread-per-connection can be fine if your
application logic is complex and you'd rather be able to block while putting
the response together, but it is generally not the best choice for
performance.

Happy hacking!

-Christian

> Thanks, Christian.
> Best regards
> David
>
>
>
> -----Original Message-----
> From: Christian Grothoff [mailto:address@hidden
> Sent: 31 January 2013 17:53
> To: David J Myers
> Cc: address@hidden
> Subject: Re: ContentReaderCallback has changed behaviour in latest 
> releases
>
> On 01/31/2013 05:43 PM, David J Myers wrote:
>> Hi Christian,
>>
>> I have a problem with the latest releases of libmicrohttpd
>>
>> Previously I was using version 0.9.7 and I have recently upgraded to
>> 0.9.20 and I have also checked this in 0.9.24
>>
>> I use a ContentReader Callback to chunk my video data
>>
>> response = MHD_create_response_from_callback (MHD_SIZE_UNKNOWN,
>>
>>                                                  1024,
>>
>> &crc, responseptr,&crcf);
>>
>> Sometime, in the callback I throw away a frame of data, so I return 
>> zero from the callback, like this.
>>
>> static ssize_t
>> crc (void *cls, uint64_t pos, char *buf, size_t max) {
>>
>> .
>>
>> .
>>
>> .
>>
>> if (NothingToDo)
>>
>> {
>>
>>      return 0; // MHD should call me back later
>>
>> }
>>
>> else
>>
>> {
>>
>>    //do stuff
>>
>> .
>>
>> .
>>
>> .
>>
>>      return no_of_bytes_processed;
>>
>> }
>>
>> }
>>
>> If I return zero from the callback I expect MHD to call me back some 
>> short time later.
>>
>> This has always worked for me in the past but now, when I return 
>> zero, from the callback, version 0.9.20 does not call me back until 
>> after a delay of around 6 seconds by which time the client has timed out.
>> Version 0.9.24 is worse and actually crashes the program with the 
>> following error:-
>>
>> Fatal error in GNU libmicrohttpd connection.c:395: API violation
>>
>>
>> Aborted
>>
>> Is it no longer allowed to return zero from a ContentReaderCallback?
>>
>>
> That depends on your threading mode.  From the documentation:
>
> /**
>    * Callback used by libmicrohttpd in order to obtain content.  The
>    * callback is to copy at most "max" bytes of content into "buf".  The
>    * total number of bytes that has been placed into "buf" should be
>    * returned.<p>
>    *
>    * Note that returning zero will cause libmicrohttpd to try again,
>    * either "immediately" if in multi-threaded mode (in which case the
>    * callback may want to do blocking operations) or in the next round
>    * if MHD_run is used.  Returning 0 for a daemon that runs in internal
>    * select mode is an error (since it would result in busy waiting) and
>    * will cause the program to be aborted (abort()).
>    *
>    * @param cls extra argument to the callback
>    * @param pos position in the datastream to access;
>    *        note that if an MHD_Response object is re-used,
>    *        it is possible for the same content reader to
>    *        be queried multiple times for the same data;
>    *        however, if an MHD_Response is not re-used,
>    *        libmicrohttpd guarantees that "pos" will be
>    *        the sum of all non-negative return values
>    *        obtained from the content reader so far.
>    * @param buf where to copy the data
>    * @param max maximum number of bytes to copy to buf (size of buf)
>    * @return number of bytes written to 'buf';
>    *  0 is legal unless we are running in internal select mode (since
>    *    this would cause busy-waiting); 0 in external select mode
>    *    will cause this function to be called again once the external
>    *    select calls MHD again;
>    *  MHD_CONTENT_READER_END_OF_STREAM (-1) for the regular
>    *    end of transmission (with chunked encoding, MHD will then
>    *    terminate the chunk and send any HTTP footers that might be
>    *    present; without chunked encoding and given an unknown
>    *    response size, MHD will simply close the connection; note
>    *    that while returning END_OF_STREAM is not technically
>    *    legal if a response size was specified, MHD accepts this
>    *    and treats it just as MHD_CONTENT_READER_END_WITH_ERROR;
>    *  MHD_CONTENT_READER_END_WITH_ERROR (-2) to indicate a server
>    *    error generating the response; this will cause MHD to simply
>    *    close the connection immediately.  If a response size was
>    *    given or if chunked encoding is in use, this will indicate
>    *    an error to the client.  Note, however, that if the client
>    *    does not know a response size and chunked encoding is not in
>    *    use, then clients will not be able to tell the difference between
>    *    MHD_CONTENT_READER_END_WITH_ERROR and
> MHD_CONTENT_READER_END_OF_STREAM.
>    *    This is not a limitation of MHD but rather of the HTTP protocol.
>    */
>
>
> So returning 0 is illegal if you're using MHD's internal select mode, 
> as we don't want applications to do busy-waiting.  It is quite 
> possible that we changed the library behavior here a bit, to perform more
stringent checking.
> There was a change related to that logic between
> 0.9.13 and 0.9.14.
>
> If you can tell me what threading mode you're using (or even have a 
> small example), that would be helpful to find out more about what's going
on.
>
> Happy hacking!
>
> Christian
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2013.0.2890 / Virus Database: 2639/6069 - Release Date: 
> 01/30/13
>

-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2013.0.2897 / Virus Database: 2639/6072 - Release Date: 01/31/13