Re: [libmicrohttpd] strange latencies using libmicrohttpd w/sendfile

[Eivind Sarto]

I am using thread per connection and poll.
Since the library always does a poll and checks POLLIN/POLLOUT before 
reading/writing the socket
I don't see why O_NONBLOCK would matter.

As for using sendfile with O_NONBLOCK, the difference is that the operation 
will not remain in the kernel
for the entire file transfer, but will return whenever the socket buffer is 
full.  The library correctly handles
both cases.
And somehow, it eliminates the long delays I get when I attempt to transfer a 
file via sendfile
and the file still has dirty buffers in the page cache.  When I have some time, 
I''ll attempt to look at
the kernel source and see why this is happening.

Maybe I will enable O_NONBLOCK on the socket only for the duration of the 
sendfile transfer.  A couple
of fcntl calls have almost no overhead.

Thanks for your feedback.

-eivind


________________________________________
From: address@hidden address@hidden On Behalf Of Christian Grothoff 
address@hidden
Sent: Tuesday, August 09, 2011 11:06 PM
To: address@hidden
Cc: Eivind Sarto
Subject: Re: [libmicrohttpd] strange latencies using libmicrohttpd w/sendfile

Hi!

There is no "strong" reason why we should not use O_NONBLOCK in some cases;
the reasons why we don't use it so far are:

1) with pthreads, we WANT to be blocking (so your patch would have to be
   setting the mode only for the non-threaded modes (thread-pool is OK))
2) takes an extra system call to set the non-blocking flag (which should not
   be necessary since we use select/poll)
3) if there is a bug where we read/write twice without a select in the middle,
   we'd notice it faster

I don't really think (2) is a good reason given your problem.  So if you make
sure to not do non-blocking in case (1) and you're pretty sure that it is not
because somehow we're in case (3) --- which would surprise me, but there are
always bugs... --- then I'd say moving to non-blocking should be fine.

Happy hacking

-Christian

On Wednesday, August 10, 2011 12:21:54 AM Eivind Sarto wrote:
> I ran into some very long HTTP latencies using libmicrohttpd and the linux
> sendfile syscall. It turned out that the problem had nothing to do with
> the library, but with the linux kernel itself (RedHat Enterprise 5.5). The
> workaround for the problem was solved by a change to the library, so I
> figured I might as well share the results with others who may also run
> into the same problem.
>
> I have an HTTP streaming server that uses libmicrohttpd.  For streaming
> most type of content, everything works just great using sendfile (via
> MHD_create_response_from_fd_at_offset).  But, when I upload the content
> from a live encoder (using PUT+DELETE) I started seeing huge latencies
> when I attempted to stream the live content.
>
> When I ran strace on the server, I noticed that the latencies came from
> either sendfile and close of the open file. The latencies were coming from
> the kernel and not the library.  It appeared to be triggered by the
> encoder writing the file (HTTP-PUT) and then the client streaming it a few
> seconds later. I am guessing that there is some interaction between
> writing the data into the page cache and writing it to the socket with
> sendfile (which does a zero-copy transfer from the page cache to the
> socket head).  And maybe the pdflush kernel daemon deciding to flush dirty
> pages to disk.
>
> libmicrohttpd does not open the connection socket in non-blocking mode. so
> I decided to add an fcntl(O_NONBLOCK) to the connection socket when the
> connection is created.
> And the delays disappeared.
>
> When streaming about 5gbps of Apple HLS Live protocol data that is being
> PUT from the encoder, the max latencies went from more than 7 seconds to
> less than 5 microseconds (measuring the latencies between the HTTP request
> and the response from the clients point of view).
>
> I am pretty sure that there is something going on in the linux kernel that
> causes the latencies. My question is:  Is there any reason why the
> connection socket should not be in O_NONBLOCK mode? And if not, could this
> be added to the library?
>
> I have a few patches I need to add to the library, so adding one more is
> not a big deal. But, I would not object if you decided to make connection
> sockets O_NONBLOCK :-)
>
> -eivind