Re: [libmicrohttpd] strange latencies using libmicrohttpd w/sendfile

[Christian Grothoff]

You're right, I forgot that we used select/poll even with threads for the 
timeouts.  So yes, non-blocking sockets should cause no difference whatsoever 
(except for the miniscule overhead of the syscall and my reason (3)), which 
are not sufficient especially given that you've seen unexplained but 
nevertheless serious performance problems without them.  I've changed the code 
to non-blocking sockets in SVN 16440.  Please let us know if you ever find out 
why the Linux kernel is so ill-behaved with blocking sockets ;-).

Happy hacking!

Christian

On Wednesday, August 10, 2011 08:03:53 PM Eivind Sarto wrote:
> I am using thread per connection and poll.
> Since the library always does a poll and checks POLLIN/POLLOUT before
> reading/writing the socket I don't see why O_NONBLOCK would matter.
> 
> As for using sendfile with O_NONBLOCK, the difference is that the operation
> will not remain in the kernel for the entire file transfer, but will
> return whenever the socket buffer is full.  The library correctly handles
> both cases.
> And somehow, it eliminates the long delays I get when I attempt to transfer
> a file via sendfile and the file still has dirty buffers in the page
> cache.  When I have some time, I''ll attempt to look at the kernel source
> and see why this is happening.
> 
> Maybe I will enable O_NONBLOCK on the socket only for the duration of the
> sendfile transfer.  A couple of fcntl calls have almost no overhead.
> 
> Thanks for your feedback.
> 
> -eivind
> 
> 
> ________________________________________
> From: address@hidden
> address@hidden On Behalf Of Christian
> Grothoff address@hidden Sent: Tuesday, August 09, 2011 11:06 PM
> To: address@hidden
> Cc: Eivind Sarto
> Subject: Re: [libmicrohttpd] strange latencies using libmicrohttpd
> w/sendfile
> 
> Hi!
> 
> There is no "strong" reason why we should not use O_NONBLOCK in some cases;
> the reasons why we don't use it so far are:
> 
> 1) with pthreads, we WANT to be blocking (so your patch would have to be
>    setting the mode only for the non-threaded modes (thread-pool is OK))
> 2) takes an extra system call to set the non-blocking flag (which should
> not be necessary since we use select/poll)
> 3) if there is a bug where we read/write twice without a select in the
> middle, we'd notice it faster
> 
> I don't really think (2) is a good reason given your problem.  So if you
> make sure to not do non-blocking in case (1) and you're pretty sure that
> it is not because somehow we're in case (3) --- which would surprise me,
> but there are always bugs... --- then I'd say moving to non-blocking
> should be fine.
> 
> Happy hacking
> 
> -Christian
> 
> On Wednesday, August 10, 2011 12:21:54 AM Eivind Sarto wrote:
> > I ran into some very long HTTP latencies using libmicrohttpd and the
> > linux sendfile syscall. It turned out that the problem had nothing to do
> > with the library, but with the linux kernel itself (RedHat Enterprise
> > 5.5). The workaround for the problem was solved by a change to the
> > library, so I figured I might as well share the results with others who
> > may also run into the same problem.
> > 
> > I have an HTTP streaming server that uses libmicrohttpd.  For streaming
> > most type of content, everything works just great using sendfile (via
> > MHD_create_response_from_fd_at_offset).  But, when I upload the content
> > from a live encoder (using PUT+DELETE) I started seeing huge latencies
> > when I attempted to stream the live content.
> > 
> > When I ran strace on the server, I noticed that the latencies came from
> > either sendfile and close of the open file. The latencies were coming
> > from the kernel and not the library.  It appeared to be triggered by the
> > encoder writing the file (HTTP-PUT) and then the client streaming it a
> > few seconds later. I am guessing that there is some interaction between
> > writing the data into the page cache and writing it to the socket with
> > sendfile (which does a zero-copy transfer from the page cache to the
> > socket head).  And maybe the pdflush kernel daemon deciding to flush
> > dirty pages to disk.
> > 
> > libmicrohttpd does not open the connection socket in non-blocking mode.
> > so I decided to add an fcntl(O_NONBLOCK) to the connection socket when
> > the connection is created.
> > And the delays disappeared.
> > 
> > When streaming about 5gbps of Apple HLS Live protocol data that is being
> > PUT from the encoder, the max latencies went from more than 7 seconds to
> > less than 5 microseconds (measuring the latencies between the HTTP
> > request and the response from the clients point of view).
> > 
> > I am pretty sure that there is something going on in the linux kernel
> > that causes the latencies. My question is:  Is there any reason why the
> > connection socket should not be in O_NONBLOCK mode? And if not, could
> > this be added to the library?
> > 
> > I have a few patches I need to add to the library, so adding one more is
> > not a big deal. But, I would not object if you decided to make connection
> > sockets O_NONBLOCK :-)
> > 
> > -eivind