Re: [Lynx-dev] lynx2.8.7pre.4

[Stefan Caunter]

On Tue, May 26, 2009 at 5:55 PM, Thorsten Glaser <address@hidden> wrote:
> David Woolley dixit:
>
>> The problem is that avoidance techniques are only short term fixes, hiding 
>> the
>> real problem, which is not a technical problem.
>
> This is one of the reasons I disagree so strongly with changing
> the DEFAULT to send no user agent header. Besides, that one really
> does make one suspicious, doesn't it? I expect the problem being
> only on a selected handfull of servers that actually use the module,
> plus most distributions would allow to file bugs against it and have
> it patched even if upstream authors don't care.

I like "suppress by default". I had to check what 2616 said.

http://tools.ietf.org/html/rfc2616#section-14.43 says

The User-Agent request-header field contains information about the
   user agent originating the request. This is for statistical purposes,
   the tracing of protocol violations, and automated recognition of user
   agents for the sake of tailoring responses to avoid particular user
   agent limitations. User agents SHOULD include this field with
   requests. The field can contain multiple product tokens (section 3.8)
   and comments identifying the agent and any subproducts which form a
   significant part of the user agent. By convention, the product tokens
   are listed in order of their significance for identifying the
   application.

SHOULD   This word, or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course.


I see two issues.

I think there is a valid reason to ignore the rfc here and suppress
the header. Servers are ignoring the rfc and have for years.
"Tailoring to limitations" cannot be interpreted to mean "don't send a
page at all". Eliminating the header means the lynx browser works
better for more users more of the time. This is the most important
consideration. Browser usage charts are a completely irrelevant metric
in this context. Does anyone here care about browser usage stats with
respect to lynx? I'm much more interested in which distributions
include lynx and if the packages work well.

The question becomes, does suppressing the header solve the banned
useragent issue? There have been several recent reports of the stock
lynx header resulting in no page being sent. Does anyone have a reason
to include the header beyond stats?

The other issue is how to present the additional option.  Users have
always been able to send whatever useragent header they like; having
the functionality appear out of the box in [O]ptions is good for
users, as would a lynx.cfg setting.

Stefan Caunter