Re: [libmicrohttpd] Epoll Connection Leak

On Wed, May 11, 2016 at 11:53 AM, Chris Penev <address@hidden> wrote:

Specifically, I believe the problem lies inside daemon.c (however, I may be wrong as I have not tested)

if (ret < (ssize_t) i)
{
/* partial read --- no longer read-ready */
...
}

because reading less bytes than the provided buffer size does not seem to mean that the next read will return EAGAIN (hence the socket could very well remain read-ready). This can be shown with the following program exhibiting the same connection leak.

#include <sys/socket.h>
#include <stdlib.h>
#include <sys/epoll.h>
#include <netinet/in.h>
#include <unistd.h>

union sockaddr_any
{
struct sockaddr addr;
struct sockaddr_in ipv4;
struct sockaddr_in6 ipv6;
};

static int xsocket(int flags)
{
int sock = socket(AF_INET, SOCK_STREAM | flags, 0);

if (sock < 0)
{
exit(EXIT_FAILURE);
}

return sock;
}

static int xepoll_create1(int flags)
{
int epfd = epoll_create1(flags);

if (epfd < 0)
{
exit(EXIT_FAILURE);
}

return epfd;
}

static void xepoll_add(int epfd, int fd, uint32_t events)
{
struct epoll_event event;
int error;

event.events = events;
event.data.fd = fd;

error = epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &event);

if (error)
{
exit(EXIT_FAILURE);
}
}

static void xreuseaddr(int sock, int on)
{
int err = setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, sizeof on);

if (err) exit(EXIT_FAILURE);
}

static void xbind(int sock, uint32_t addr, in_port_t port)
{
union sockaddr_any any;
int err;

any.ipv4.sin_family = AF_INET;
any.ipv4.sin_addr.s_addr = htonl(addr);
any.ipv4.sin_port = htons(port);

err = bind(sock, &any.addr, sizeof any.ipv4);

if (err) exit(EXIT_FAILURE);
}

static void xlisten(int sock, int backlog)
{
int err = listen(sock, backlog);

if (err) exit(EXIT_FAILURE);
}

static void slurp(int fd)
{
static unsigned char buffer [4096];
ssize_t nbytes = sizeof buffer;

while (nbytes == sizeof buffer)
{
nbytes = read(fd, buffer, sizeof buffer);
}

if (!nbytes) close(fd);
}

int main(void)
{
int sock = xsocket(SOCK_NONBLOCK);
int epfd = xepoll_create1(0);

xreuseaddr(sock, 1);
xbind(sock, 0x7f000001, 8080);
xepoll_add(epfd, sock, EPOLLIN | EPOLLET);
xlisten(sock, 32);

for (;;)
{
struct epoll_event ev;

if (epoll_wait(epfd, &ev, 1, -1) != 1)
{
continue;
}

if (ev.data.fd == sock)
{
int conn = accept(sock, 0, 0);
xepoll_add(epfd, conn, EPOLLIN | EPOLLET);
continue;
}

slurp(ev.data.fd);
}

return EXIT_SUCCESS;
}

Note however, that if the while loop inside slurp is changed to

while (nbytes > sizeof buffer)
{
nbytes = read(fd, buffer, sizeof buffer);
}

Closed connections are properly terminated.

On Tue, May 10, 2016 at 6:16 PM, Chris Penev <address@hidden> wrote:
Hello everyone. When I use epoll (specifically external select), I can create scenarios where the microhttpd library fails to detect client disconnects. Assuming microtthpd is listening on 127.0.0.1:8080 and a client does

$ echo -n GET | nc 127.0.0.1 8080

The microtthpd library receives two epoll events. Hence an strace shows something along the lines of

epoll_wait(...)
recvfrom(..., "GET", ...)
recvfrom(..., "")
close(...)

However, if the client does something like this

$ { sleep 1; echo -n GET; } | nc 127.0.0.1 8080

Then in the above case microhttpd receives only one epoll event. Hence, strace shows something along the lines of

epoll_wait(...)
recvfrom(..., "GET", ...)

Even futher calls to epoll_wait will not return additional events, and microhttpd will never attempt a second recvfrom, and never figure out that the client is gone.

I believe that because the epoll interface used by microhttpd is the edge triggered one (which should be the more efficient one), microhttpd needs to continue calling read until it receives EAGAIN, as documented under "man epoll".

Otherwise, the connection ends up in the CLOSE-WAIT state. Note that in the above example, none of the client handlers have been called either, so the application cannot do anything about it either.

Lastly, I've attempted to subscribe to the mailing list, but have not received a confirmation email, so please note that I may not be able to read a reply that replies to the mailing list only.

Sincerely,
Chris P

From:	Chris Penev
Subject:	Re: [libmicrohttpd] Epoll Connection Leak
Date:	Wed, 11 May 2016 12:00:28 -0700