Re: [Lynx-dev] Viewing Dave's Trailer page with Lynx 2.8.7dev.5: error

[Chuck Houpt]

Hi Patrick,

It looks like the website is browser sniffing, and rejecting anything  
with 'libwww' or 'libcurl' in the User-Agent field. Here are some  
tests with curl that show the blocking of libwww/libcurl, but not  
'foobar':

$ curl -A 'libwww' -I http://www.davestrailerpage.co.uk/
HTTP/1.1 406 Not Acceptable

$ curl -A 'libcurl' -I http://www.davestrailerpage.co.uk/
HTTP/1.1 406 Not Acceptable

$ curl -A 'foobar' -I http://www.davestrailerpage.co.uk/
HTTP/1.1 200 OK

Possibly the site is trying to deter rogue robots and scrapers, but  
filtering based on the User-Agent field won't be very effective, since  
it is easily changed (for example from the Lynx options page).

Cheers - Chuck
  http://chuck.habilis.net


On Apr 16, 2009, at 4:06 AM, Patrick Vervoorn wrote:

> Hello,
>
> My apologies for sending this 'out of the blue', since I did not  
> subscribe to the lynx-dev list.
>
> However, I'm seeing a problem which is rather Lynx-specific, since  
> viewing the same URL, on the same machine, but using Lynx, works  
> rather ok.
>
> The full story is in the email below which I sent to Dave, but since  
> w3m works fine, I'm not sure it's a problem with the site...
>
> Regards,
>
> Patrick.
>
> ---------- Forwarded message ----------
> Date: Sun, 12 Apr 2009 17:01:51 +0200 (CEST)
> From: Patrick Vervoorn <address@hidden>
> To: address@hidden
> Subject: Viewing Trailer page with Lynx: error
>
> Hi there,
>
> When I try to view your site after the latest move with Lynx, I  
> invariably get the following error-message:
>
> (This is when viewing both the main site at: http://www.davestrailerpage.co.uk/ 
>  but also happens when I try to view one of the sub-pages directly,  
> like: http://www.davestrailerpage.co.uk/about.html)
>
>
>                                 Not Acceptable
>
>   An appropriate representation of the requested resource / could  
> not be
>   found on this server.
>
>   Additionally, a 404 Not Found error was encountered while trying  
> to use
>   an ErrorDocument to handle the request.
>
>
>
> When using something like w3m, it is displayed. Using a graphical  
> browser like Firefox also works fine.
>
> Is there anything you can do from your side to be able to view your  
> Trailer page using Lynx?
>
> Best regards,
>
> Patrick.
>
>
> _______________________________________________
> Lynx-dev mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/lynx-dev