Re: [libmicrohttpd] connection statistics

[Eivind Sarto]

How useful various stats will be can certainly be discussed.
I believe being able do show the rate of connections, requests, bytes 
transferred
(and possibly also number of active connections and requests) will be very 
helpful
when trying to diagnose load issues on a server with 10s of thousands of active 
connections.

In order to show the rate of connections being made you have to take multiple 
samples of the total number of
connections (and take the delta and divide by time).  I don't think there is an 
easy way to collect the total number of
HTTP connections, outside of the library itself.
It should be a relatively trivial to add a counter or two to the library.  
Creating/destroying connections within the library
is already race-free.  I don't mind adding a simple patch to the library.

I just thought there might be others who possibly wanted something similar, or 
maybe already have added some statistics
to libmicrohttpd.
I will be happy to share with others any such changes/patches that I may make.

As for adding epoll to libmicrohttpd, I don't see it as being all that useful.  
If you run the daemon threaded mode, each thread
will be polling a single fd.  I have run oprofile on the system while serving 
over 10k connections and close to 10 gbps of network
load.  The kernel polling code is not significant at all.

-eivind  


________________________________________
From: address@hidden address@hidden On Behalf Of Christian Grothoff 
address@hidden
Sent: Saturday, February 11, 2012 3:20 AM
To: address@hidden
Subject: Re: [libmicrohttpd] connection statistics

The real issue with this kind of performance monitoring is that for
multi-threaded applications it is very hard to give you a "sound"
number.  Also, I don't see how this would be helpful for the application
that is running at the time.

If the system operator wants this information, it is pretty easy to get
a snapshot: lsof and netstat will show you how many connections the
application has open and what the overall TCP state looks like for the
OS.  So for server status monitoring, why not use these standard tools?

Now, if you actually have a very large number of mostly keep-alive
connections, there are two possible answers: shorter timeout (so that
they go away), or you're yet another person who'd like to see ePoll
support for MHD (which is not that easy to add, hence it is not likely
that I'll find the time to do this anytime soon).  Looking at how many
select/poll calls you're doing and how long they take (for example,
using 'strace -c -e trace=select,poll') might be a good idea here.

In summary, I'm not sure I see a need for this kind of
performance-monitoring support to be integrated with MHD, as the OS
already gives you this and more high-quality information easily.

My 2 cents!

Happy hacking!

Christian

On 02/11/2012 12:40 AM, Eivind Sarto wrote:
> The completion callback handler is called whenever an HTTP request has 
> completed.
> At that point there is no way of knowing if the connection remains open.
>
> -eivind
> ________________________________________
> From: address@hidden address@hidden On Behalf Of Keith Mendoza address@hidden
> Sent: Friday, February 10, 2012 3:32 PM
> To: libmicrohttpd development and user mailinglist
> Subject: Re: [libmicrohttpd] connection statistics
>
> If you're intent is to see the number of active connections at the
> moment I would say that this is something that you should be able to
> do from within your application. This might even prove useful to you
> at some future time in handling streaming of the video data. I can't
> provide any use cases, but I'm getting the "it might be useful" itch.
>
> If I understand it correctly, the completion handler is called when
> the HTTP connected between your application and the client is closed.
> So, if the keep-alive is still there wouldn't that mean that the
> connection is still open and the completion callback should not be
> called?
>
> On Fri, Feb 10, 2012 at 3:17 PM, Eivind Sarto<address@hidden>  wrote:
>> I am using libmicrohttpd for a video streaming project.
>> The HTTP server load can be quite high with a large number of requests per 
>> seconds and a high network bandwidth,
>> but the HTTP connections are mostly keep-alives.
>>
>> In order to troubleshoot load related problems, it can be difficult to get 
>> an idea of what is going on a given server.
>> I want to add some kind of server status monitoring and I would like to be 
>> able to display the number of active
>> connections/requests and total connections/requests (among other things).
>>
>> I can collect total and active number of requests from counters in the 
>> callback to the default handler and the completion handler.
>> However, I cannot find a way to get the total and active number of 
>> connections.  The only one who can keep an accurate count of
>> the number of connections is the internals of the library.
>>
>> Does anyone see any value to adding some statistics/counters to 
>> libmicrohttpd that can returned via an API?
>>
>> Or, is there something I am overlooking that I could do to display the 
>> number of active connections and total connections created?
>>
>> -eivind
>>
>>
>
>