Re: [libmicrohttpd] infinite loop even when connection is closed by client in the create_response_from_callback when max < buffer size

[Christian Grothoff]

On Friday, January 21, 2011 01:03:32 pm Leandro Santiago wrote:
> I'm using MHD_USE_THREAD_PER_CONNECTION | MHD_USE_DEBUG option flags and no
> (0) timeout to start the daemon. About chunked encoding, are you talking
> about the header "Transfer-Encoding: chunked"? I'm sending the data as a
> normal file, a sequence of bytes. The only difference is these bytes don't
> exist in disk, they are generated in runtime. I'm not using this header.
> But even when I add it, the behavior is the same. What I do is generate
> something with size less than 360000 bytes, copy them into the buf via
> memcpy and return the size of this data.
> 
> What's the meaning of the max parameter?

How many bytes you are allowed to place into the buffer at most (you're 
allowed to provide fewer).
 
> > In any case, looking at the code (there are two places in connection.c
> > where
> > we call the content reader callback, both with the format
> > "response->crc", the
> > 'max' argument should always be either block_size (unchunked) or
> > block_size -
> > 12 (chunked encoding), unless you're at the end of the file (which for
> > MHD_SIZE_UNKNOWN you'd never be).
> 
> When everything runs fine (for example, using mplayer as client) every
> block has the correct size and max also has this same value.

It is odd that the choice of client has an impact here.  Chunking is something 
the server can choose to do, the client has little to say.  Can you check if 
you're using HTTP 1.0 with one client and HTTP 1.1 with the other?  That's 
seem to be the only difference that the choice of client should really have.
 
> > So a value of 16372 is rather strange (359988 OTOH should be something
> > you ought to have to deal with...).  Could you answer the above
> > questions and maybe also supply a stack trace with the values of the
> > "response" and "connection" structs on the stack upon the time of the
> > call?
> > 
> > Finally, in terms of being called "forever", you may want to add a
> > timeout to
> > your connections (option to MHD_start_daemon).
> 
> Now I discovery what is the number 16372:
> It's (2^14)-12 = 16384 - 12 = 16372
> 
> So I think I'm sending chunked data and the maximum size of data that the
> web browsers support is 2^14 bytes.

But the web browser doesn't get to choose in HTTP.  And in fact, neither does 
TCP or anything else, this is really a pure choice of the HTTP server.  Now, 
MHD will use chunked encoding IFF:
1) you have HTTP 1.1 (could be the case here for the browser!) AND
2) you did not specify a message size (check!) AND
3) you do not set "Connection: close" (you probably didn't do this either).

So my best guess is that mplayer goes with HTTP 1.0 and does not reuse TCP 
streams whereas your browser goes for chunked encoding.  Now, that should cut 
of 12 bytes, but I still don't see where the 2^14 would come from.  Still, 
adding a "Connection: close" header to your response (and not getting chunked 
encoding) should be an easy first test / workaround.

 
> So I have one question: when I pass to the create_response_from_callback
> the buffer size, does it mean I need to return a data with this exactly
> size?

No, you should not have to.  MHD will simply reserve a buffer of that size and 
allow you to pass chunks of up to that size, but smaller chunks should work 
fine as well.
 
> So I reduced the size passed to this function (and changed some internal
> logic about the size of buffer), now everything is working fine :-)

Well, 360k buffers/connection (and no timeouts!) seems to be a bad idea anyway 
(too much memory footprint/connection), but what I find strange is that this 
should have just worked regardless (minus the 12 bytes for chunked encoding if 
you really needed exactly 360k byte-chunks).

Happy hacking,

Christian

> Thanks a lot.
> 
> > Happy hacking,
> > 
> > 
> > Christian
> > 
> > On Wednesday 19 January 2011 13:37:37 Leandro Santiago wrote:
> > > Hello to all. I'm streaming videos via http using libmicrohttpd.
> > > 
> > > To do it, I'm using the MHD_create_response_from_callback, using a
> > 
> > callback
> > 
> > > with the interface:
> > > 
> > > static ssize_t video_generator(void* cls, uint64_t pos, char* buf,
> > > size_t max);
> > > 
> > > When I call MHD_create_function_from_callback, I'm passing the value
> > 
> > 360000
> > 
> > > as the block_size and MHD_SIZE_UNKNOWN as size;
> > > 
> > > The problem is sometimes the max value isn't 360000, but a lower value
> > > (16372).
> > > 
> > > As I can't work with this low value, I ask, in the beginning of
> > > video_generator:
> > > 
> > > if (max < 360000) return 0;
> > > 
> > > The problem is thereafter this function (video_generator) is called
> > > everytime, indefinitely, passing max as 16372 and the "pos" value as 0.
> > > Even if I close the connection (closing the client, for example), the
> > > video_generator callback continues indefinitely, with the same values
> > 
> > (max
> > 
> > > = 16372 and pos = 0). So I need to kill the process.
> > > 
> > > For example, the above behavior happens when I use any webbrowser as
> > > client, but doesn't happen when I use mplayer (which supports http
> > > protocol).
> > > 
> > > What this function does is: everytime it's called, it sends one frame
> > > to the client.