Re: [Bug-wget] http server responding with 416 but file was not transferred completely

[Josef Moellers]

On 18.09.2017 16:48, Tim Rühsen wrote:
> Hi Josef,
> 
> pushed your patch - thanks for your contribution.

Welcome

> BTW, the example from your original post doesn't show that wrong server
> behavior any more. Maybe the server got fixed meanwhile !?

I guess so. Probably a glitch.
That's why I posted the little server that couldn't ;-)

Josef

> On 09/14/2017 07:47 PM, Josef Moellers wrote:
>> On 14.09.2017 17:06, Tim Rühsen wrote:
>>> On 09/14/2017 12:11 PM, Josef Moellers wrote:
>>>> On 14.09.2017 10:12, Tim Rühsen wrote:
>>>>> On 09/14/2017 09:53 AM, Josef Moellers wrote:
>>>>>> Hi,
>>>>>>
>>>>>> We have seen (at least) one server who has
>>>>>> Accept-Ranges: bytes
>>>>>> in his header but, upon receiving a request with
>>>>>> Range: bytes=23068672-
>>>>>> responds with
>>>>>> HTTP/1.1 416 Requested Range Not Satisfiable
>>>>>> although the file was not transferred completely!
>>>>>>
>>>>>> wget then claims that
>>>>>> The file is already fully retrieved; nothing to do.
>>>>>>
>>>>>> E.g.
>>>>>> run
>>>>>>   wget https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO
>>>>>> the, after a couple of MB, abort the transfer and then continue the
>>>>>> download:
>>>>>>   wget --continue
>>>>>> https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO
>>>>>>
>>>>>> Maybe the check in src/http.c:
>>>>>> 3821   if (statcode == HTTP_STATUS_RANGE_NOT_SATISFIABLE
>>>>>> 3822       || (!opt.timestamping && hs->restval > 0 && statcode ==
>>>>>> HTTP_STATUS_OK
>>>>>> 3823           && contrange == 0 && contlen >= 0 && hs->restval >= 
>>>>>> contlen))
>>>>>>
>>>>>> should be changed and any HTTP_STATUS_RANGE_NOT_SATISFIABLE with an
>>>>>> incomplete file should show something like
>>>>>>
>>>>>> "download continue failed, file incomplete"
>>>>>
>>>>> Well, that would be ok for this special server.
>>>>>
>>>>> Normally 416 together with a server timestamp matching the file's
>>>>> timestamp means that the file is complete (as far as the client can
>>>>> judge from HTTP).
>>>>>
>>>>> IMO, if the server is broken (or misbehaves) then the server should be
>>>>> repaired. Except it is a very common misbehavior. In which case we could
>>>>> consider a work-around on the client side.
>>>>>
>>>>
>>>> So I humbly propose the attached patch.
>>>> I tried to create a pull request, but got a 403.
>>>
>>> Thanks for the patch - I'll test it in the next days.
>>
>> I have attached a simple webserver that simulates the error:
>> when a request with a Range comes in, it replies with 416 and also
>> returns an unsanely huge Content-Length. You'll need glib2 and
>> microhttpd for it to build.
>>
>> I was able to reproduce the issue with this server and check that the
>> patch fixes it by causing wget to retry (until --retries is exhausted).
>>
>>> BTW, we currently work on Wget2 where we have a related issue, if you
>>> like to take a look at it: https://gitlab.com/gnuwget/wget2/issues/278
>>
>> I'll do that.
>>
>> Josef
>>
>