[libmicrohttpd] single-threaded daemon, multiple pending requests, responses batched

[Frank Ch. Eigler]

Hi -

One of the programs I work on uses libmicrohttpd to serve a web
protocol.  It uses vanilla single-threaded select mode.

Some of the requests require timetaking operations on the order of
seconds.  A web browser might make multiple such requests in parallel
to the MHDish process (on separate sockets).  The problem is that
responses to the requests are batched in a way, so nothing is sent out
until all requests have finished processing.

What seems to be happening is that the MHD_run function recognizes
that the various incoming http connections are all ready for reading,
does the recvfrom() to fetch the HTTP instruction, then dispatches
each one, one at a time, to the MHD app for handling.  Each callback
enqueues proper response documents (say 30K each), and returns
MHD_YES.  But MHD_run does not try to send those responses right away.

I suspect this may be because MHD_run_from_select changes
event_loop_info state of the connection from _READ to _WRITE at the
bottom of the while (pos ... next) loop.  Even if the state changes,
the loop still just proceeds to the next connection, instead of trying
to send the available data via the pos->write_handler().

The effect is that each MHD_run's worth of output takes until the next
invocation of MHD_run in order to actually transmit output.  This
increases the TTFB latency of the *all* responses to the sum of the
processing time of *all* concurrently-arriving requests.

I realize that single-threaded mode is not a great fit for this
application.  But is there anything we can hope for in terms of
tweaking the state machine, or use MHD options or funky flow-control
MHD calls, or reentrant MHD_run* calls to improve on the latency?

Thanks!

- FChE