Re: [libmicrohttpd] Complete request on fd becoming ready

[Kenneth Mastro]

See inline comments, below.

Ken

On Mon, Mar 27, 2017 at 9:04 AM, Hein-Pieter van Braam <address@hidden> wrote:

Hi!

Thank you for your reply, I think my original question wasn't very well
posed. Let me try to rephrase it. 

In the epoll + threadpool configuration of MHD, MHD itself uses an
epoll() type loop to service fds that are ready to send or receive
traffic from clients. I/O is non-blocking and when MHD is waiting on
the kernel to finish sending traffic back to the client, or waiting on
traffic from the client, the fd goes back into the wait list of epoll.

What I'm hoping to achieve is to receive traffic from my backend in a
similar manner. So I'm hoping for something like this:

* MHD mainloop accept()'s a new connection
* MHD pushes this new connection to a thread on its threadpool
* MHD runs the request handler for the connection
* Request handler parses the HTTP request and constructs a request in
the binary protocol.
* Request handler opens a non-blocking connection to the backend
server.
< .. magic .. >
* The request gets sent to the backend server once the connection to
the backend becomes ready for writing.
< .. more magic .. >
* Once the listening FD for the backend connection becomes ready,
create a response object and send it to the client to have MHD then use
its normal async loop to service the client.

I'm not entirely sure what 'magic' you need.  I may not understand your problem well enough, though.  It sounds like you want to use MHD's suspend/resume functionality to allow MHD to use epoll internally to service connections, and then feed it responses when the response is ready.  I have not used MHD's suspend/resume code personally (I use the 'thread per connection' mode), but I know many people have.

From what I recall, the gist is:

* You get a request from the caller.  That causes MHD to call your code to process it.

* You do what you need to get things going, and then 'suspend' the connection.

* This allows MHD to use that thread to continue to process incoming requests.

* Once the result is ready, you call the 'resume' function to allow you to do stuff with that connection again.

* At this point I THINK MHD calls your callback again, thus giving you the option to queue a response.  (I.e., you don't just queue it from any old thread.  You have to wait for one of MHD's threads to service you for that particular connection.)

I'm a bit hazy on the details and/or logistics of the above, but I think that's about right.  I personally can't help more with that, but I think searching the mailing list archive might be helpful.  I've seen messages roll through here in the past couple years that talk about this.  Maybe somebody with more experience in this area can chime in.

What I'm thinking of doing now is to do something like this in the
request handler:

* Send the parsed request to a separate thread that has its own epoll()
loop, along with struct MHD_Connection *connection.
* Return MHD_OK
* In the separate thread, deal with the request like normal, when the
backend socket becomes ready for reading construct the MHD_Response and
call MHD_queue_response (connection, MHD_HTTP_OK, response);
* Destroy the response object

Now, I'm not entirely sure if this will work for a variety of reasons.
I don't know if it's safe toe call MHD_queue_response() in this manner
for instance.

See my above comment.  It sounds like you want to use the suspend/resume functionality and you're just wondering how to get it to work.  Valid question.  I'm not sure how robust the example code for suspend/resume is.  (I remember thinking it wasn't impressively clear and/or wasn't really available at the time, but I only gave it cursory attention.  The mailing list archive may be of more help.)

 

I was also wondering if there was a way of combining these loops so I
don't have to do it this way. Maybe if external select does what I
think it does I could use that as my programs' main loop instead of
MHD's, but I don't know if that will actually be any better.

Fair enough.  I don't know if you can service other things in an MHD external epoll.  (I don't see why not, but I have never used MHD's external select stuff.)  If you have the horsepower (and it won't take much), keeping the loops separate may be a bit simpler.  I.e., skip the external select.  Let MHD do its thing and you do your thing.  You have the problem of getting the data back to the caller either way and will still need to use suspend/resume since you shouldn't block the MHD threads.  (Or just use thread per connection, of course.  You can do whatever you want in those threads because MHD doesn't need them to service other connections with them.)

 

On Mon, 2017-03-27 at 08:33 -0400, Kenneth Mastro wrote:
>
> If traffic/request volume is sufficiently low, the simplest thing to
> do is use MHD's 'thread per connection' mode and just do the back-end
> I/O in that thread.  Send the request, wait for the response, and
> send the result back out MHD to the caller.  Just make sure the back-
> end is thread-safe and/or the MHD side waits for a request to finish
> before sending another (even if it's simply by using a mutex).  It's
> hard to know what to suggest without a lot more info.

This is more or less what I'm doing now. Doing blocking I/O to the
backend service in a thread per connection.

> In short - this sounds a lot more like a threading problem than any
> problem you might have with MHD.

It's not so much a 'problem' as that I'm trying to figure out how to
optimize my program so it can be fully asynchronous.

>
>
> On Sun, Mar 26, 2017 at 8:44 PM, Hein-Pieter van Braam <address@hidden>
> wrote:
> > Hello all,
> >
> > I am writing an MHD application where the MHD HTTP server acts as a
> > proxy between a HTTP REST API and a different binary protocol on
> > the
> > backend. My MHD application opens a new TCP connection to the
> > backend,
> > translates the request, and waits for an answer.
> >
> > The waiting on an answer part would normally be blocking, and
> > reading
> > the MHD manual I can't come up with a way of using async io here.
> > What
> > would an MHD application look like that creates response objects
> > from
> > (another) polling loop, and where do I even place this loop? 
> >
> > I have been thinking of a variety of convoluted things with
> > threads,
> > then I figured it'd probably be better to ask the experts :)
> >
> > Thanks!
> >
> > - HP
> >
> >
>
>