Re: [libmicrohttpd] Odd Stop Behavior

[Kenneth Mastro]

Hi Christian,

Thanks for the quick reply.

And erk!  Sorry about the OS thing.  I'm on Linux.  My development environment is a stock install of Ubuntu 12.04 LTS VM on a Windows 7 PC.  (I doubt the VM thing matters at all, but figured I'd mention it just in case.)

Here's my MHD_start_daemon code, nothing earth shattering ( I assume that's where the per-connection timeout would be set):
-----------------------
myDaemon = MHD_start_daemon(MHD_USE_THREAD_PER_CONNECTION, // one thread per connection
                                WEB_TEST_PORT,                 // port to listen on
                                nullptr,                       // accept policy callback
                                nullptr,                       // extra arg to accept policy callback
                                &connectionCallbackC,          // main 'connection' callback
                                this,                          // extra argument to the 'connection' callback
                                MHD_OPTION_NOTIFY_COMPLETED,   // specifies that the next arg is the callback to call when the connection is done
                                &requestCompletedCallbackC,    // the callback to call when the connection is done
                                this,                          // extra arg to the callback when the connection is done
                                MHD_OPTION_CONNECTION_LIMIT,   // specifies that the next arg is the max number of simultaneous connections
                                (unsigned int)20,              // the number of permitted simultaneous connections
                                MHD_OPTION_END);               // no more options in the arg list
----------------------

So - no timeout changes for MHD.  I haven't changed the default timeout for the kernel, so I'm guessing that 115 (awfully close to 120, i.e., 2 minutes) is something else?  I agree completely that it's likely some kind of timeout, though.

I don't change any thread timeouts at all (although I have some other threads in my application that I set to real-time priority for some audio processing, but I can't imagine how that would adversely affect MHD).

If I didn't mention it before, I did trace this down to the 'join' of the thread in 'close_all_connections'.  I assume 'MHD_join_thread_' is actually pthread_join since I'm on Linux.  My gut was telling me that somehow the thread isn't being told to stop as part of the shutdown process and is eventually just timing out somewhere inside MHD (or via TCP as you suggested).

Before I noticed this problem, I was running with 'MHD_USE_SELECT_INTERNALLY' with a thread-pool, but you had suggested in a previous post (a few weeks ago) that that won't work properly with 'comet-like' requests unless I do the suspend/resume functionality.  That's perfectly fine and good, but it seems worth mentioning that I didn't notice this problem when using that thread mode.  From looking at the 'close_all_connections' code in daemon.c, I can see why the behavior could be different.

Anyway - I'll play around with the HAVE_LISTEN_SHUTDOWN option, see if that makes any difference.  Failing that, I'll see if I can create a test case.  That make take a couple days, though - lots of code to strip out to keep it simple.  What should I do once I get a concise test case ready to go?  Send the code to the mailing list?


Thanks again for a great library - it's perfect for what I'm doing and I've really enjoyed using it.

Ken

On Fri, Jun 6, 2014 at 5:52 PM, Christian Grothoff <address@hidden> wrote:

Dear Kenneth,

The first thing you should provide is the name of the operating system
you are on (if non-Linux, that might by itself be a big hint).

Regardless, given that MHD does terminate after 115s suggests that some
timeout is involved.  Did you, by chance, set the per-connection timeout
to like 115-120 seconds?  Or is that your kernel's TCP timeout?  Again,
that's just for diagnostics -- there clearly still is an issue as that
timeout should not matter for shutdown.


When MHD is stopped, it should terminate existing ongoing requests
immediately (client gets a closed TCP socket); this is usually done
by calling 'shutdown()' on the TCP socket if HAVE_LISTEN_SHUTDOWN
(which should unblock the pthread), or by writing to a pipe.  Again,
what happens exactly depends a lot on your thread mode, MHD options
and kernel.

However, shutdown issues like this are usually easy to debug by
attaching a debugger and seeing where that thread hangs around,
so if you could provide a testcase I'm sure we can find a solution
rather quickly.


Happy hacking!

Christian

On 06/06/2014 08:50 PM, Kenneth Mastro wrote:
> I'm seeing an odd behavior when I shut down the daemon (MHD_stop_daemon).
> The shutdown hangs for about 115 seconds, then it finally terminates
> cleanly.
>
> I'm using the 'thread per connection' mode, which may be relevant.  After
> digging into microhttpd's code, I found that it was hanging on a thread
> join during 'close_all_connections'.
>
> I began doing some heavy debugging, and determined that I don't have any
> lingering connections that I never finished with.  I even went so far as to
> print out the thread PIDs in both microhttpd and my code to make sure I saw
> the requests that were being handled by the hanging thread.  Nothing fancy
> - just some css and/or image files being served up (about 25 requests or
> so).  Through this debugging, I'm nearly certain that I'm ending the
> connection/request callback every time (I also did a global
> increment/decrement of requests, and it gets back to 0 every time - Ergo,
> I'm not holding onto the thread.).
>
> Some other useful info:
> * When I set the maximum number of connections/threads to 1 (instead of 20)
> when I start the daemon, it does not hang when I stop the daemon - it just
> ends immediately, as expected.
>
> * After I start shutting down, if I make ANOTHER request from the browser
> (e.g.., refresh the page), the daemon stops immediately and cleanly.
>
> * After I load the page the first time (and it makes a bunch of requests to
> the server, causing 1 or more threads to be used), if I wait a while (e.g.,
> a minute) and then stop the server, it shortens the time it takes the
> thread to terminate.  I.e., if I wait 65 seconds, it only takes 70 seconds
> for the server to stop instead of 115.  If I wait longer than 115 seconds,
> it will end immediately, as expected.
>
> * If I make NO requests (i.e., I just start, then stop a few seconds
> later), it terminates cleanly.  The 115 second delay only occurs after I've
> made a request to the server from the browser and then try to stop it.
>
> * I'm using a fairly recent version: 0.9.35
>
>
> I came across some vaguely related issues regarding stopping the daemon:
> -----------------
> I saw some old archived messages about a problem that existed in
> microhttpd, but that was several years ago, but it sounds like it was a
> full-on deadlock:
> https://lists.gnu.org/archive/html/libmicrohttpd/2009-10/msg00013.html
>
> Also saw this, which may be similar to what I'm dealing with (only it
> sounds like this was solaris specific, and I'm on Linux):
> http://lists.gnu.org/archive/html/libmicrohttpd/2011-09/msg00024.html
>
> Also, this issue feels similar as well:
> https://gnunet.org/bugs/view.php?id=1901
> ------------------
>
> It seems like there's been some trouble in this area in the past, so I
> figured it was worth asking about.
>
> So - any idea if I'm doing something wrong?  I'm at a bit of a loss since
> my code doesn't have any control over the internal threads.  I'm happy to
> try anything in my application and/or modify microhttpd to test, but I'm
> not sure where to focus my efforts.
>
>
> Thanks much,
> Ken
>