Re: [libmicrohttpd] strange latencies using libmicrohttpd w/sendfile

[Christian Grothoff]

Hi!

There is no "strong" reason why we should not use O_NONBLOCK in some cases; 
the reasons why we don't use it so far are:

1) with pthreads, we WANT to be blocking (so your patch would have to be
   setting the mode only for the non-threaded modes (thread-pool is OK))
2) takes an extra system call to set the non-blocking flag (which should not
   be necessary since we use select/poll)
3) if there is a bug where we read/write twice without a select in the middle,
   we'd notice it faster

I don't really think (2) is a good reason given your problem.  So if you make 
sure to not do non-blocking in case (1) and you're pretty sure that it is not 
because somehow we're in case (3) --- which would surprise me, but there are 
always bugs... --- then I'd say moving to non-blocking should be fine.

Happy hacking

-Christian

On Wednesday, August 10, 2011 12:21:54 AM Eivind Sarto wrote:
> I ran into some very long HTTP latencies using libmicrohttpd and the linux
> sendfile syscall. It turned out that the problem had nothing to do with
> the library, but with the linux kernel itself (RedHat Enterprise 5.5). The
> workaround for the problem was solved by a change to the library, so I
> figured I might as well share the results with others who may also run
> into the same problem.
> 
> I have an HTTP streaming server that uses libmicrohttpd.  For streaming
> most type of content, everything works just great using sendfile (via 
> MHD_create_response_from_fd_at_offset).  But, when I upload the content
> from a live encoder (using PUT+DELETE) I started seeing huge latencies
> when I attempted to stream the live content.
> 
> When I ran strace on the server, I noticed that the latencies came from
> either sendfile and close of the open file. The latencies were coming from
> the kernel and not the library.  It appeared to be triggered by the
> encoder writing the file (HTTP-PUT) and then the client streaming it a few
> seconds later. I am guessing that there is some interaction between
> writing the data into the page cache and writing it to the socket with
> sendfile (which does a zero-copy transfer from the page cache to the
> socket head).  And maybe the pdflush kernel daemon deciding to flush dirty
> pages to disk.
> 
> libmicrohttpd does not open the connection socket in non-blocking mode. so
> I decided to add an fcntl(O_NONBLOCK) to the connection socket when the
> connection is created.
> And the delays disappeared.
> 
> When streaming about 5gbps of Apple HLS Live protocol data that is being
> PUT from the encoder, the max latencies went from more than 7 seconds to
> less than 5 microseconds (measuring the latencies between the HTTP request
> and the response from the clients point of view).
> 
> I am pretty sure that there is something going on in the linux kernel that
> causes the latencies. My question is:  Is there any reason why the
> connection socket should not be in O_NONBLOCK mode? And if not, could this
> be added to the library?
> 
> I have a few patches I need to add to the library, so adding one more is
> not a big deal. But, I would not object if you decided to make connection
> sockets O_NONBLOCK :-)
> 
> -eivind