Re: [libmicrohttpd] Server Loads

[Christian Grothoff]

Yes, I did notice this when benchmarking the "src/examples/benchmark"
program using 'ab' as the client.  On my system, it was at somewhat
higher numbers (we did >= 45k/second on loopback), but obviously the
limits are going to be hardware-dependent.  What I could observe was
that a few requests showed rather high latencies (seconds, as also shown
by your MAX), while 99% had "normal" latencies of a few ms.

The "benchmark" program also prints a distribution between MHD accepting
a connection and MHD completing it (ab measures client-side, so
benchmark.c was written to measure server-side).  There, the MAX was
still higher, but not as dramatic.

I did go over MHD's code and could not find any cause why MHD should
behave "unfairly" among clients.  I suspect that maybe some kernel
parameters might need to be tweaked here (one certainly needs to play
with TIME_WAIT options at this connection rate), but I've not really
spent much time trying to figure out what is exactly going on.

In any case, I am pretty sure that it is unlikely to be an issue in your
application logic (as benchmark can exhibit similar behavior), and I'm
also reasonably convinced that the issue is not purely in MHD --- most
likely it is either the Linux kernel or (less likely) an odd interaction
between MHD and the kernel at high speeds.  If you find out what is
going on or how to fix it, please do let us know ;-).

happy hacking!

Christian


On 08/23/13 20:30, Derrick Bradbury wrote:
> Hi all...
> 
> Just wanting to see if anyone has noticed this before.
> 
> We're running an application in an envirionment and it seems to handle
> requests up to about 7500/second...
> 
> Soon as we start creeping over that strange things happnen. Requests
> that are at max allowed to run for 6ms take > 10 seconds, connections
> timeout, and random server reboots (this one may be a xen thing).
> 
> I have a little script that collects the reqeusts and then  does some
> analysis on it:
> 
> AVG:0.0229191
> MIN: 0.000010
> MAX: 7.403612
> CNT:83403
> 0ms:36307
> 1ms:10847
> 2ms:5135
> 3ms:3357
> 4ms:2606
> 5ms:2119
> 6ms - Warning:1775
> Timeout:21257
> Err:4352
> 
> Our MINIMUM response time at about 12,000 requests/second is 0.00001
> seconds... and the maximum is 7.4 seconds. And the high number of
> TIMEOUTS is concerning, as the app shouldn't allow that.
> 
> Basically the app takes a URL and submits it to a 3rd party database to
> be classified. I have a 6ms timeout so there really shouldn't be
> anything > 6ms, but I'm getting a lot of timeouts.
> 
> 
> Some bits of  code:
> 
> 
>         struct MHD_Daemon *daemon;
>         daemon = MHD_start_daemon (MHD_USE_SELECT_INTERNALLY
>                         | MHD_USE_EPOLL_LINUX_ONLY | MHD_USE_EPOLL_TURBO,
>                         options.port,
>                         NULL, NULL, &answer_to_connection, NULL,
>                         MHD_OPTION_THREAD_POOL_SIZE, (unsigned int)
> options.threads,
>                         MHD_OPTION_CONNECTION_LIMIT, (unsigned int)
> 12000, MHD_OPTION_END);
> 
> 
> 
> THe code where I have a timeout at 6ms:
>     try {  
>       if (!ready) throw 997;
>       iurl = MHD_lookup_connection_value (connection,
> MHD_GET_ARGUMENT_KIND, options.key);
>            
>       if (!iurl) throw 997;
>       if (options.strip_path && strlen(iurl) > 35) iurl=strip_path(iurl);
>       if (strcmp("facebook.com",(char*)iurl) == 0 && options.client) {
>         page = "76\n78\n80\n";
>       }
>       else {
>         boost::thread cls(classify_url,iurl,boost::ref(rp),internalid);
>        
> if(!cls.timed_join(boost::posix_time::milliseconds(options.timeout))) {
>           cls.interrupt(); 
>           timeout_counter++;
>           page=(options.client)?"998\n":"{\"c\": [998]}";
>         }  
>         else
>           page = rp.c_str();
>       }
>     }
>     catch(...) {
>       bad_counter++;
>       page=(options.client)?"997\n":"{\"c\": [997]}";
>     }
> 
>