Re: [libmicrohttpd] Problem with HTTP/1.1 chunked packets when connection is about to close

[Evgeny Grin]

Привет, Яна!

Actually this behaviour is not a bug.
Chunked encoding produces binary larger output and makes heavier system 
and network load, so MHD tries to avoid chunked output if possible.

If you provide response size as the first argument for 
MHD_create_response_from_callback() MHD will never use chunked encoding.

If you provide "-1" (MHD_SIZE_UNKNOWN) as the size then MHD will follow 
the next logic:
* if connection is "keep-alive" then only choice is chunked encoding and 
chunked encoding is used,
* if connection have to be closed after sending response then end of 
response can be indicated by end of connection so "heavier" chunked 
encoding is not used.

This logic most probably will be re-considered soon as latest HTTP/1.1 
specification forbids responses without both encoding and content-size 
(and ending connection without specifying content size is unreliable).

However I discovered bug related to chunked encoding (when application 
forcibly sets "Transfer-Encoding: chunked") and "Connection: close". It 
will be fixed in the next MHD release.

--
Wishes,
Evgeny

On 25.04.2021 19:45, Iron Bug at gmail.com wrote:
> Greetings.
>
> I use libmicrohttpd in my project and I found one bug in a very specific 
> situation with chunked messages.
>
> The scheme I use is keep-alive connection for transferring some files (chunked 
> content, HTTP/1.1 pipelining). When transfer is over, connection is closed by 
> client request.
>
> I create responses with MHD_create_response_from_callback and the HTTP header "Transfer-Encoding: chunked" is 
> automatically added to packet headers. But when the transfer is over and the last file requested, client sends 
> "Connection: close". Theoretically, by HTTP standard, the server should process the request and then close 
> the socket. And it does so, it sends the chunked data and closes the connection. But in the response to the last 
> request it loses the "Transfer-Encoding: chunked" header and the client fails to accept the wrong formatted 
> HTTP packet. If I manually add "Transfer-Encoding: chunked" to the last HTTP response, it works fine.
>
> I think the problem hides in this code piece:
>
>        file src/microhttpd/connection.c, line 1376, in function 
> have_chunked_upload:
> -----------------------------------------------------------------
>        if ( (MHD_NO != keepalive_possible (connection)) &&
>             (MHD_str_equal_caseless_ (MHD_HTTP_VERSION_1_1,
>                                       connection->version) ) )
>        {
>          if (NULL == have_encoding)
>          {
>            must_add_chunked_encoding = true;
>            connection->have_chunked_upload = true;
>          }
>         ...
> -----------------------------------------------------------------
>
> The connection is marked as non keep-alive at the moment after "Connection: close" and the content is still chunked. But the flag is not set and 
then the error occurs.
> Sincerely, Yana A. Kireyonok aka Iron Bug

OpenPGP_signature
Description: OpenPGP digital signature