Re: [libmicrohttpd] shutdown of listen socket does not work on solaris 10

[Christian Grothoff]

On 09/16/2011 01:53 PM, Frank Meier wrote:
> hi
>
> I tried to use libmicrohttpd on solaris 10, and my simple programm only
> starting and stopping the daemon (see below) did hang in.
> after some investigation (see debug code) I found, that the shutdown
> call in daemon.c fails, since the daemon socket is not a connected
> fullduplex but a listening socket.

Ah, I see.  Well, on *BSD and GNU/Linux (and AFAIK even W32), 'shutdown' 
of a listen socket causes a corresponding select/poll call to unblock. 
 I didn't know Solaris disagreed here.  I see where this comes from, 
but shutdown can be used to close a socket 'for reading' and on a listen 
socket, you read for incoming connections (see usage of listen sockets 
in select), so if I close a listen socket for reading, then that means 
that I won't be accepting on it anymore.  At least, that's what most 
POSIX-implementations do...

> this means the daemon thread keeps
> hanging in select() and main hangs in pthread_join() (see pstack
> output). problem is since the socket doesn't get closed, select does not
> interrupt. After adding CLOSE (fd) after the point of SHUTDOWN () the
> programm did work. It looks like solaris and linux behave differently
> when shutdown is called on a listening socket. But on the other hand I'm
> not sure if it is correct to call shutdown in the first place, since the
> man page states "The shutdown() call causes all or part of a full-duplex
> connection on the socket associated with sockfd to be shut down" and the
> listening socket does not have a connection.
>
> I think the socket should be closed with CLOSE before calling
> pthread_join() and use SHUTDOWN only on connection sockets.

Well, that would seem to solve the problem, but would introduce a race. 
 With 'close', another thread might open a file or socket and MHD's 
might then select/poll/accept on that unrelated socket.  So 'close' is 
actually incorrect (admittedly, it is a "solution" that will work in at 
least 99.999% of all actual runs, but still not correct -- EBADF is not 
an errno value I like to see, ever).

Also, other solutions in this context, some involving pipes, signals for 
threads (pthread_kill) etc. were rejected because they are not supported 
on some platforms and/or are inefficient.  'shutdown' was specifically 
used because it seemed to be a clean way to avoid the problems with the 
various alternatives. Except, it seems, on Solaris. Yuck.

So I'm not sure I have a good answer for you at this point. Suggestions 
welcome.

Happy hacking!

Christian

> regards, Frank
>
>
> /*test programm*/
> struct MHD_Daemon* mhdDaemon = 0;
> mhdDaemon = MHD_start_daemon (MHD_USE_THREAD_PER_CONNECTION, mhdPort,
> NULL, NULL, &answer_to_connection, NULL, MHD_OPTION_END);
> if (mhdDaemon)
> {
> MHD_stop_daemon (mhdDaemon); // <-here it hangs
> mhdDaemon = 0;
> }
>
> /*debug code in daemon.c*/
> --- daemon/daemon.c (revision 16873)
> +++ daemon/daemon.c (working copy)
> @@ -2477,7 +2477,11 @@
> daemon->worker_pool[i].shutdown = MHD_YES;
> daemon->worker_pool[i].socket_fd = -1;
> }
> - SHUTDOWN (fd, SHUT_RDWR);
> + int res = SHUTDOWN (fd, SHUT_RDWR);
> + int tmpErrno = errno;
> + printf("shutdown res:%d %s\n", res, strerror(tmpErrno));
> +
> + CLOSE (fd);
>
> ->output with debug code:
> shutdown res:-1 Transport endpoint is not connected
>
> /*pstack output -> damon thread is in select, main hangs in pthread join*/
> $ pstack 16802
> 16802: ./test.bin
> ----------------- lwp# 1 / thread# 1 --------------------
> feee9365 lwp_wait (2, 80478d8)
> feee2c1f _thrp_join (2, 0, 8047918, 1) + 5a
> feee2d9e pthread_join (2, 8047918, feeec7e0, fef9665d) + 2b
> fef96777 MHD_stop_daemon (8060d40, 0, 0, fee63301) + 12b
> 0805080a main (1, 8047988, 8047990) + 6f
> 080506ac _start (1, 8047a88, 0, 8047a93, 8047abe, 8047b78) + 60
> ----------------- lwp# 2 / thread# 2 --------------------
> feee8da5 pollsys (fed5ed50, 1, 0, 0)
> fee93afa pselect (4, fed5ef30, fed5eeb0, fed5ee30, 0, 0) + 18e
> fee93df0 select (4, fed5ef30, fed5eeb0, fed5ee30, 0, 0) + 82
> fef95ba4 MHD_select (fec50200, fec50200, fed5efec, feee59b9, 8060d40, 0)
> + 130
> fef96248 MHD_select_thread (8060d40) + 18
> feee59b9 _thr_setup (fec50200) + 4e
> feee5ca0 _lwp_start (fec50200, 0, 0, fed5eff8, feee5ca0, fec50200)