Re: [libmicrohttpd] Processing large binary data in chunks?

[Christian Grothoff]

Hi!

Given that both of you are essentially asking for the same thing, I've taken
the time to actually implement the feature you're asking for in SVN 30402
(https://gnunet.org/svn/libmicrohttpd/).

The code is simple, well-documented but essentially untested (thus feedback
would be very welcome).  Essentially, instead of going for the busy waiting
you just call MHD_suspend_connection, and when you're ready for more, you
again call MHD_resume_connection.  This should work for uploads and
downloads.

Jared, to answer your question more specifically, MHD only reads as much
from
the socket as it has buffer space available for; so there is no problem with
a "hard" error here, this is just about not having MHD go crazy calling
application callbacks that simply are not ready to deliver (or process)
data
yet.


Happy hacking!

Christian


On 10/23/2013 10:19 PM, Adam Almos Homolya wrote:
> I would like to stream large amount of data that is being read
> asynchronously so that I can't ensure that when libmicrohttpd is calling
> the reader callback any byte is available for read without blocking. The
> documentation says returning zero from the callback will result in a busy
> wait which is not an option. Can you give any advise how to work around
> this problem?

On 10/23/2013 09:58 PM, Jared Cantwell wrote:
> We haven't tested the extremes of using a thread per connection and
> "blocking" 10MB at a time, but I was wondering if there was an obvious
> reason why that wouldn't work (assuming we trickle enough to prevent socket
> timeouts)?  It seems like the MHD event loop might pull as much as it can
> off the socket, which means that if we haven't completely emptied the
> buffer an error could occur?  Or does MHD first try to only pull enough
> data off the socket for the amount of space available in the buffer?
> 
> 
> On Wed, Oct 23, 2013 at 11:28 AM, Christian Grothoff <address@hidden>wrote:
> 
>> This is not possible right now.  I agree that we should fix this,
>> I'll try to find the time to implement this soon. (We will need a
>> new API call to suspend/resume processing of a connection.)
>>
>> Happy hacking!
>>
>> Christian
>>
>> On 10/23/2013 04:27 PM, Jared Cantwell wrote:
>>> Is it possible to use libmicrohttpd to transfer large amounts of binary
>>> data (e.g. 10GB) incrementally using a thread pool model?  By
>> incrementally
>>> I mean that after we receive 10MB we'd like to process the 10MB
>> internally
>>> before allowing more data to be transferred to the web server-- we don't
>>> have enough memory to receive the whole 10GB first.  Using
>>> MHD_USE_THREAD_PER_CONNECTION we can achieve this by blocking until we
>>> process each 10MB chunk.  However, for scaling reasons we'd prefer to use
>>> the thread pool model, but I can't find a way to "block" acknowledgement
>> of
>>> processing some data without synchronously blocking one of the threads in
>>> the pool.  It looks like the POST request handlers are close to what I
>>> need, except my data is not form-encoded.
>>>
>>> Any suggestions would be greatly appreciated.
>>>
>>> Thanks,
>>> Jared
>>>
>>
>>
>