Re: [libmicrohttpd] Problems with latency and getting "stuck"

[Christian Grothoff]

On 04/05/14 02:03, maurice barnum wrote:
> Hi.
> 
> I'm working on a project where I want to use libmicrohttpd to handle
> incoming HTTP connections that are then forwarded to a ZeroMQ server.

It would be helpful if you would mention which MHD version you are using.

> Three problems I'm encountering:
>  * latency is bad due to a significant delay between when I wake up
> a connection with MHD_resume_connection and when MHD_run calls the
> corresponding handler callback

What do you consider a 'significant delay' here? In your data plot, it
seems you're concerned with each millisecond or a few microseconds, but
maybe I missed something.

>  * each run of MHD_run will accept several incoming connetions but
> only "retire" one of the connections I've resumed.

"retire" or "resume"?  I am pretty sure MHD_run is perfectly happy to
tear down multiple connections during one invocation.  And yes, as
MHD_run may accept fresh connections before processing 'resume' events,
this may increase latency, especially given that you're using a single
thread for all processing.  If you are concerned with
milli/micro-seconds, I wonder why you are not using a thread pool.

>  * eventually, everything stops:  I call zmq_poll with a timeout of -1
> (MHD_get_timeout() returned 0), but the epoll fd never signals read-in
> even when new connections come in.

You have set a connection limit of 25. If you suspended 25 connections,
the accept FD (and all 25 suspended connections) will be out of the
epoll set, and your server grinds to a halt until you resume a connection.

> My event loop listens on a ZeroMQ socket and the epoll fd returned from
> MHD.  The loop looks basically like (pseudo-code):
> 
>    while true:
>      MHD_run(md)
>      timeout = MHD_get_timeout() / 1000
>      if not timeout:
>        timeout = -1
>      zmq_poll(items, len(items), timeout)

I hope you're somehow having the MHD epoll socket in the zmq_poll set here.

>      if items[0].revents:
>         # this is the zeromq socket
>         if items[0].revents and ZMQ_POLLIN:
>           # loop elided
>           m = zmq_read()
>           context = find_context(m.identity)
>           context.response = m.data
>           MHD_resume_connection(context.mc)
> 
> On the callback side, I send a request over the ZeroMQ socket and
> then suspend the connection until my event loop picks up a response.
> 
> Curiously, I found that if I didn't call MHD_run() each time through
> the loop, my server would get wedged immediately, which is why I don't
> call it iff zmq_poll signals that the epoll fd is readable.

I can't quite see why this would happen, except if the timeout is what
MHD is waiting on, but that should not relate to an immediate halt (but
just connection timeouts not happening properly).

> This is how I init the daemon:
> 
>     const unsigned mhdflags =
>         MHD_USE_DEBUG |
>         MHD_USE_SUSPEND_RESUME |
>         MHD_USE_EPOLL_LINUX_ONLY;
> 
>     DaemonPtr result(MHD_start_daemon(
>                 mhdflags,
>                 httpPort,
>                 NULL, NULL, // accept callback
>                 handlerCallback, self,
>                 MHD_OPTION_NOTIFY_COMPLETED, connectionComplete, self,
>                 MHD_OPTION_CONNECTION_TIMEOUT, 5,
>                 MHD_OPTION_CONNECTION_LIMIT, 25,
>                 MHD_OPTION_END),
>             MHD_stop_daemon);
> 
> Any ideas on where to look/debug is appreciated.  I've attached a
> digested trace of my debug output that shows the behavior.

I'd try with a connection limit of 1 first, that should simplify what
happens, and you should encounter certain problems immediately instead
of only after 25 'suspended' connections.

Happy hacking!

Christian

signature.asc
Description: OpenPGP digital signature