RE: [libmicrohttpd] recv MSG_DONTWAIT and EAGAIN

[Eivind Sarto]

I looked at the library source a bit closer, and as you say, recv(MSG_DONTWAIT) 
should never return EAGAIN since the poll/select makes sure
there is data on the socket head before the recv is called.

I if start the server with  MHD_USE_SELECT_INTERNALLY instead of MHD_USE_POLL 
it runs perfectly.  I never see any EAGAINs.
Maybe there is something going on with the poll logic?
I don't think it is an OS issue.  I am running linux (redhat enterprise 5.5).
It is possible that my code corrupts something that causes the poll logic to 
mess up, but the symptom is so constant and predictable that I doubt it.  

I'll try and debug it since I can easily reproduce the problem with just a 
single (iPod touch) client.

Note:  I have also seen the EAGAIN trigger an mhd_panic() in 
connection:build_header_response().
This has only happened in a few cases.  I added a debug/printf right before the 
panic and the connection state was MHD_CONNECTION_CLOSED.
That sort of indicates that the EAGAIN from the recv can trigger a problem in 
the connection state processing.

I'll keep you posted.

-eivind

________________________________________
From: address@hidden address@hidden On Behalf Of Christian Grothoff 
address@hidden
Sent: Friday, March 18, 2011 5:30 AM
To: address@hidden
Cc: Eivind Sarto
Subject: Re: [libmicrohttpd] recv MSG_DONTWAIT and EAGAIN

Given that we defined MSG_DONTWAIT to 0 if it was not defined by the platform, 
clearly the logic was never supposed to depend on non-blocking operations. I 
think the DONTWAIT was added as a "conservative" flag ("just in case"), after 
all, the calls are all guarded by 'select' and 'poll'.

So this makes the situation here interesting:

1) select/poll should only allow us to get to recv if data is ready (and note

that we're using select/poll even with threads). So technically the

DONTWAIT should not have mattered --- dead code should go, right?

2) On the other hand, the DONTWAIT may have prevented worse -- a blocking

read of the server due to some other error that would stop all

connections was avoided at the expense of dropping one.

Clearly neither is good. Furthermore, we got an EAGAIN which, while unexpected 
(see above) it by itself usually not a reason to close a connection --- except 
if our select/poll is broken and we're about to go into a busy-waiting spin 
cycle, maybe then closing the connection is a good idea after all?

Finally, based on your report, the issue is tricky for you to reproduce 
("timing related"), so I don't have an easy testcase to settle who is to blame 
(after all, theoretically your application could have read data from the socket 
between MHD's select/poll and recv call, or corrupted memory, or ... --- not 
that I'm saying this is even likely, my money is on poll+threads still having 
some deeper trouble that I just cannot find by looking at the code).

Given all this, I've decided to follow your suggestion and remove MSG_DONTWAIT 
(from both recv and send calls) for now, the primary reason being that it was 
never used on *all* platforms and so this should give us consistent behavior, 
if nothing else, and, by design, it should do no harm. [SVN 14672]

I'll keep the code to close connections on EAGAIN as well, after all, that 
should not have happened and now definitively should never happen in the first 
place.

Naturally, I'd be very interested to hear if someone now suddenly observes 
blocking behavior on a recv call (obviously hard to observe on 
pthread-per-connection...) even more so if they can produce a testcase to 
demonstrate it.

Happy hacking,

Christian

On Wednesday, March 16, 2011 07:25:05 pm Eivind Sarto wrote:

> I am using libmicrohttpd-0.9.8 for a prototype video streaming application

> and I am seeing some unexpected error messages from the library when I run

> the application on a very fast server. I think the behavior is timing

> related, because it does not occur when I run the same code on a slightly

> slower server, or if I enable an MHD_OPTION_URI_LOG_CALLBACK handler on

> the fast server.

>

> In daemon.c:recv_param_adapter(), RECV(MSG_DONTWAIT) will frequently return

> EAGAIN, and this causes connection.c:do_read() to call

> connection_close_error().

>

> Is this the right logic?

> Calling RECV(MSG_DONTWAIT) should expect EAGAIN if there is no data queued

> on the socket head. It is not really and error that should close the

> connection.

> What do you think?

>

> I am running the library with flags = MHD_USE_THREAD_PER_CONNECTION |

> MHD_USE_POLL, so for debug I decided to try and remove the MSG_DONTWAIT

> flag. And the application would then run without any errors.

>

> Note: I do not have any control over the client. I am streaming live video

> to an Apple handheld device.

>

> -eivind