Re: [libmicrohttpd] a strange thing when send call returns -1 in send_cls()

[Christian Grothoff]

On 11/01/2014 07:04 AM, Taehwan Weon wrote:
> I am testing reliability of my caching proxy based on MHD, which opens 80
> port.
> It means anyone can use it to reach any web server.
> Most of incoming requests seems either incomplete or odd.

Most!? Or do you mean "some"?

> (I found a client opened a HTTP session and sent only a part of headers,
> but no body data.)
>
> 1. To avoid spurious EAGAIN, I add a counter into MHD_Connection, which
> counts continuous errors. If a threshold reaches, ECONNRESET set. EAGAIN
> seems caused by buffer full on client side. I am not sure what is the
> client's purpose.

That's odd, TCP flow control should take care of that and simply not
present the (busy) connection as writable to the server userspace.
Still, if you are right, this suggests an easy way to
test for that condition -- by implementing a client that doens't
read the response and by sending a 'large' reply that won't fit
into the buffers. Have you tried this to reproduce the issue?

> 2, To avoid infinite wait to receive remaining part of a client's data, I
> need timeout mechanism.
>     I am not sure where is the best location to handle timeout. MHD or my
> caching proxy would be the candidate.

I would say that virtually every MHD production site should set 'some'
timeout for MHD.  Note that you can set a custom timeout for a
particular connection, i.e. when you know that a specific connection
will take way longer than usual.

Happy hacking!

Christian
> 
> 
> ------
> weon
> 
> 2014-11-01 4:23 GMT+09:00 Christian Grothoff <address@hidden>:
> 
>> MHD should tolerate EAGAIN (and just retry), as the kernel can generate
>> that 'spuriously' (i.e. without good reason). Are we really terminating
>> the connection on EAGAIN? Is this reproducable? Can you produce a stack
>> trace for a case where we get an EAGAIN and then close the connection?
>>
>> If we (as we should) ignore EGAIN and try again, this should be OK, and
>> might at worst be a kernel-level performance issue (useless context
>> switching).
>>
>> Happy hacking!
>>
>> Christian
>>
>> On 10/31/2014 07:39 AM, Taehwan Weon wrote:
>>> Hi,
>>>
>>> I applied the final MHD update to my project.
>>> Still some strange result.
>>>
>>> Below is my MHD log generated by my custom send_param_adaptor()
>>>
>>> 15:31:58/000 {16138} [ INFO]  message repeated 8042083 times, MHD: socket
>>> 483 - 12165 bytes tried, but ret = -1, errno = 11
>>> 15:32:09/000 {16138} [ INFO]  message repeated 8534594 times, MHD: socket
>>> 483 - 12165 bytes tried, but ret = -1, errno = 11
>>> 15:32:20/000 {16138} [ INFO]  message repeated 8542227 times, MHD: socket
>>> 483 - 12165 bytes tried, but ret = -1, errno = 11
>>> 15:32:31/000 {16138} [ INFO]  message repeated 8546356 times, MHD: socket
>>> 483 - 12165 bytes tried, but ret = -1, errno = 11
>>> 15:32:32/889 {16132} [ INFO]  MHD: socket 479 - 385 bytes tried, but ret
>> =
>>> -1, errno = 104
>>> 15:32:32/889 {16138} [ INFO]  MHD: socket 483 - 12165 bytes tried, but
>> ret
>>> = -1, errno = 11
>>> 15:32:32/889 {16132} [ INFO]  MHD: Failed to send data: Connection reset
>> by
>>> peer
>>> (intentionally omitted)
>>> 15:32:34/648 {16138} [ INFO]  MHD: socket 483 - 12165 bytes tried, but
>> ret
>>> = -1, errno = 11
>>> 15:32:34/648 {16132} [ INFO]  MHD: Failed to send data: Connection reset
>> by
>>> peer
>>> 15:32:34/648 {16138} [ INFO]  MHD: socket 483 - 12165 bytes tried, but
>> ret
>>> = -1, errno = 11
>>> 15:32:45/000 {16138} [ INFO]  message repeated 8033311 times, MHD: socket
>>> 483 - 12165 bytes tried, but ret = -1, errno = 11
>>>
>>> As you see, there are too many erroneous repeats on the socket even in
>>> error case. (errno is 11 (EAGAIN))
>>> In such a case, I don't know what to do.
>>>
>>>
>>> Thanks in advance.
>>>
>>>
>>>
>>> ------
>>> weon
>>>
>>>
>>> 2014-10-30 0:53 GMT+09:00 Christian Grothoff <address@hidden>:
>>>
>>>> On 10/29/2014 08:42 AM, Taehwan Weon wrote:
>>>>> While testing our web-server (based on MHD 0.9.38), sometimes, I found
>>>>> heavy cpu load.
>>>>> It caused by infinite send_cls() calls because send call in send_cls()
>>>>> returned -1 with errno of 0.
>>>>>
>>>>> I read and read again send() manual page, but there is no mention about
>>>> the
>>>>> case that send() returns -1 with errno of 0.
>>>>
>>>> I agree, that's why our logic doesn't quite expect that. But, nobody
>>>> said libc/kernel isn't allowed to have bugs ;-).  So my suggestion would
>>>> be this: I modify MHD to handle this (by closing the connection, don't
>>>> see what else we can do), and you should please take it up with CentOS,
>>>> asking them to fix that send() call. (What might be relevant these days:
>>>> is this using GNU libc or diet libc / ulibc?)
>>>>
>>>>> My current environment is:
>>>>>      OS Kernel :  2.6.32-358.11.5.el6.x86_64 (CentOS)
>>>>>      MHD options:  MHD_USE_POLL_INTERNALLY |
>>>> MHD_USE_THREAD_PER_CONNECTION|
>>>>> MHD_USE_DEBUG;
>>>>>
>>>>>
>>>>> Any help will be highly appreciated.
>>>>
>>>> I've patched MHD in SVN 34391, please let us know if that helps (note
>>>> that IF the kernel does return -1 on send() with no errno, the new logic
>>>> should simply close the connection).
>>>>
>>>> Happy hacking!
>>>>
>>>> Christian
>>>>
>>>>
>>>
>>>
>>
>>
> 
>

signature.asc
Description: OpenPGP digital signature