Re: [Bug-wget] A strange issue when using wget on ssh remote host.

[Micah Cowan]

On 07/27/2012 07:02 AM, Hongyi Zhao wrote:
> But, when I ssh'ed to the remote HPC of my university and then run the
> same command as mentioned0 above, I'll meet the following issue:
> 
> -------------
> address@hidden:~> wget -c http://registrationcenter-
> download.intel.com/akdlm/irc_nas/2699/
> parallel_studio_xe_2011_sp1_update3_intel64.tgz
> --2012-07-28 02:50:45--  http://registrationcenter-download.intel.com/
> akdlm/irc_nas/2699/parallel_studio_xe_2011_sp1_update3_intel64.tgz
> Resolving registrationcenter-download.intel.com... 204.2.133.19,
> 204.2.133.64
> Connecting to registrationcenter-download.intel.com|204.2.133.19|:80...
> connected.
> HTTP request sent, awaiting response... 200 OK
> Length: unspecified [text/html]
> Saving to: `parallel_studio_xe_2011_sp1_update3_intel64.tgz'
> 
>     [ <=>                                   ] 194         --.-K/s   in
> 0s
> 
> 2012-07-28 02:50:46 (37.0 MB/s) -
> `parallel_studio_xe_2011_sp1_update3_intel64.tgz' saved [194]
> --------------
> 
> As you can see, in the later case, the destination file was incorrectly
> identified as "unspecified [text/html]" file, so wget exit immediately.

Not quite accurate. Unspecified refers to the length, and means that the
server didn't send the length of the file. This, in turn, means that
wget has no way of knowing whether a server disconnect happens because
there was a problem, or if the server is just disconnecting because it's
done sending the file. It would appear that the server disconnected
after sending only 194 bytes of content, so wget assumed it was done.

I tried twice on my local box, with wget versions 1.13.4, and 1.10.2,
and they both went fine. I can't really tell you why you got the
behavior you did, but I do notice that "dig
registrationcenter-download.intel.com" gives me different answers every
time I run it (looks to be managed with Akamai (my employer), so IPs
returned will most likely vary by location and load).

I also tried running both versions of wget on the specific IP address
that seemed to be getting used by your remote server (though it also
seems to have been used on your local box at least once), by setting it
in /etc/hosts.

There are several potential things that could be going on, most of which
are hard to diagnose.

 - You could be using a version of wget that was built without large
file support - it's conceivable that such a wget could have difficulties
related to the file's size. This seems an unlikely explanation, given
that the server also told you that the content was [text/html], which
wouldn't be explained by this situation.

 - Your remote wget could be going through a "transparent proxy", that
inappropriately changed the real response from the server.

 - The remote server might have been temporarily misconfigured. This
seems unlikely if you ran the remote and local tests at around the same
time, since you got the same IP addresses for the --spider checks, at least.

 - The remote server might have been partially broken, and returned a
different response to one machine, perhaps due to the origin IP address.

You might want to look at the 194-byte html file that gets downloaded on
the "broken" system: it may be a message from a proxy, or the server,
telling you there was a problem.

-mjc