[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[libmicrohttpd] RE: MHD_handle_connection() performance issue
From: |
Eivind Sarto |
Subject: |
[libmicrohttpd] RE: MHD_handle_connection() performance issue |
Date: |
Tue, 22 Mar 2011 21:06:23 -0400 |
I going to give some feedback to my own posting :-)
I think the solution is that MHD_handle_connection() should not set the POLLIN
flag for the listen/server socket.
It should change the code to the following:
fd = con->daemon->socket_fd;
p[1].fd = fd;
// p[1].events |= POLLIN; /*OLD*/
p[1].events = 0; /*NEW*/
Even if pollfd.events=0, pollfd.revents=POLLHUP when the daemon socket is shout
down. And that is what we care about, right?
We don't want to know if there is any input data on the daemon socket.
I looked at the linux kernel source and it will set pollfd.revents=POLLHUP when
the socket is closed, regardless of the value
of pollfd.events. Not sure if every OS behaves this way, but maybe someone
who knows other OS can provide some feedback.
Ultimately, it would probably be better (faster) to only poll a single fd (the
connection socket) and find some other way to
detect if the server is shutting down, but this change makes a huge difference
in system load with lots of connections.
-eivind
________________________________________
From: address@hidden address@hidden On Behalf Of Eivind Sarto address@hidden
Sent: Tuesday, March 22, 2011 5:16 PM
To: address@hidden
Subject: [libmicrohttpd] MHD_handle_connection() performance issue
I am using libmicrohttpd to serve a large number of video players that is using
a "smooth" streaming format.
That means the video stream is divided into separate chunks and that the client
can decide to jump to a
faster or slower encoding of the video stream on any chunk boundary. The
length of a video chunk is
typically several seconds (<= 10 sec).
(If you have 1000 clients all playing videos in 10 second chunks, you will have
100 requests/sec)
I wrote a client simulator so I can do some load testing without buying a whole
bunch of client devices.
The server gets extremely busy as crank up the number of clients. I am running
the server with
MHD_USE_THREAD_PER_CONMHDNECTION | MHD_USE_POLL
The reason for this appears to be that MHD_handle_connection() will poll both
the connection socket and
the listen/server socket. And every time there is a new connection, all the
active connections will return
from poll, because they are also polling the listen/server socket.
It gets pretty awful as you try and crank up 1000 simulated streams. The cpus
are almost completely saturated with
each connection/thread polling again and again, each time a new connection is
made.
I made a debug version of MHD_handle_connection() that only polls the
connection socket, and I can run several
1000 video clients with almost no cpu load.
But, each connection thread can no longer detect if the server is shutting down.
Any suggestion or feedback about what could be done to improve this situation?
Note: I am using the latest svn version with the fixes to MHD_poll() for
MHD_USE_THREAD_PER_CONMHDNECTION.
-eivind