Re: [libmicrohttpd] connection statistics

[Will Bryant]

FWIW, Eivind isn't the only one who would use this - I needed it for my app to 
produce app monitoring statistics.  (This sort of thing is a requirement for 
devops - we only have one infrastructure component in production that is not 
capable of instrumenting itself, and we're writing it out.)

I ended up adding the following patch:

unsigned int MHD_count_active_connections(struct MHD_Daemon *daemon) {
        struct MHD_Connection *connection;
        unsigned int result = 0;
        if (pthread_mutex_lock(&daemon->cleanup_connection_mutex) != 0) return 
0;
        connection = daemon->connections_head;
        while (connection) { connection = connection->next; result++; }
        if (pthread_mutex_unlock(&daemon->cleanup_connection_mutex) != 0) 
return 0;
        return result;
}

Although I don't like O(n) methods, for my app's use case this is acceptable.  
Of course, if I were rolling this patch into the library, I'd make it a 
MHD_DaemonInfo option to MHD_get_daemon_info, but as a patch it's easier to 
maintain as a separate method.

Counting requests would be more complicated, but you can much more easily count 
request starts and request completions using the provided callbacks, so if that 
is an acceptable definition of active requests the difference between those and 
the connection count can give you the number of inactive connections.

Yes, the OS can count active connections.  But I don't have to rewrite my 
monitoring code and test harness code to work on three OSs with their own 
netstat formats, so it's much easier to get it from the app itself.


On 12/02/2012, at 00:20 , Christian Grothoff wrote:

> The real issue with this kind of performance monitoring is that for 
> multi-threaded applications it is very hard to give you a "sound" number.  
> Also, I don't see how this would be helpful for the application that is 
> running at the time.
> 
> If the system operator wants this information, it is pretty easy to get a 
> snapshot: lsof and netstat will show you how many connections the application 
> has open and what the overall TCP state looks like for the OS.  So for server 
> status monitoring, why not use these standard tools?
> 
> Now, if you actually have a very large number of mostly keep-alive 
> connections, there are two possible answers: shorter timeout (so that they go 
> away), or you're yet another person who'd like to see ePoll support for MHD 
> (which is not that easy to add, hence it is not likely that I'll find the 
> time to do this anytime soon).  Looking at how many select/poll calls you're 
> doing and how long they take (for example, using 'strace -c -e 
> trace=select,poll') might be a good idea here.
> 
> In summary, I'm not sure I see a need for this kind of performance-monitoring 
> support to be integrated with MHD, as the OS already gives you this and more 
> high-quality information easily.
> 
> My 2 cents!
> 
> Happy hacking!
> 
> Christian
> 
> On 02/11/2012 12:40 AM, Eivind Sarto wrote:
>> The completion callback handler is called whenever an HTTP request has 
>> completed.
>> At that point there is no way of knowing if the connection remains open.
>> 
>> -eivind
>> ________________________________________
>> From: address@hidden address@hidden On Behalf Of Keith Mendoza address@hidden
>> Sent: Friday, February 10, 2012 3:32 PM
>> To: libmicrohttpd development and user mailinglist
>> Subject: Re: [libmicrohttpd] connection statistics
>> 
>> If you're intent is to see the number of active connections at the
>> moment I would say that this is something that you should be able to
>> do from within your application. This might even prove useful to you
>> at some future time in handling streaming of the video data. I can't
>> provide any use cases, but I'm getting the "it might be useful" itch.
>> 
>> If I understand it correctly, the completion handler is called when
>> the HTTP connected between your application and the client is closed.
>> So, if the keep-alive is still there wouldn't that mean that the
>> connection is still open and the completion callback should not be
>> called?
>> 
>> On Fri, Feb 10, 2012 at 3:17 PM, Eivind Sarto<address@hidden>  wrote:
>>> I am using libmicrohttpd for a video streaming project.
>>> The HTTP server load can be quite high with a large number of requests per 
>>> seconds and a high network bandwidth,
>>> but the HTTP connections are mostly keep-alives.
>>> 
>>> In order to troubleshoot load related problems, it can be difficult to get 
>>> an idea of what is going on a given server.
>>> I want to add some kind of server status monitoring and I would like to be 
>>> able to display the number of active
>>> connections/requests and total connections/requests (among other things).
>>> 
>>> I can collect total and active number of requests from counters in the 
>>> callback to the default handler and the completion handler.
>>> However, I cannot find a way to get the total and active number of 
>>> connections.  The only one who can keep an accurate count of
>>> the number of connections is the internals of the library.
>>> 
>>> Does anyone see any value to adding some statistics/counters to 
>>> libmicrohttpd that can returned via an API?
>>> 
>>> Or, is there something I am overlooking that I could do to display the 
>>> number of active connections and total connections created?
>>> 
>>> -eivind
>>> 
>>> 
>> 
>> 
> 
>