bug#16220: url-http.el: Not conforming to HTTP spec

[Jarosław Rzeszótko]

Hi,

There are more issues besides just the extra "\r\n", the url-http.el
library is super confusing. I read the documentation for url-retrieve,
which makes doing a HTTP request look really straightforward, and then
I spent several hours finding out what the hell happens. For example,
the documentation for url-retrieve says:

"The variables `url-request-data', `url-request-method' and
`url-request-extra-headers' can be dynamically bound around the
request; dynamic binding of other variables doesn't necessarily take
effect."

The problem is that url-http.el sets a lot of headers by default that
can not be overwritten in any other way then dynamically overshadowing
some variables. For example, all connections are keep-alive by
default, which is confusing in itself already, futhermore you can't
just pass "Connection" . "close' in url-request-extra-headers, because
url-http.el joins the default and "extra" headers instead of merging
the "extra" ones and having them overwrite the default ones. So try do
something like this:

(let ((url-request-method "GET")
      (url-request-extra-headers '(("Connection" . "close"))))
  (url-retrieve-synchronously "http://www.google.com/";))

And what is sent is this:

GET / HTTP/1.1
Connection: keep-alive
...
Connection: close

Which again isn't valid HTTP and the behaviour of the HTTP server in
this case is undefined and implementation specific. The only way to
workaround this is doing this:

(let ((url-http-attempt-keepalives nil)
      (url-request-method "GET")
      (url-request-extra-headers '(("Connection" . "close"))))
  (url-retrieve-synchronously "http://www.google.com/";))

Which is only clear by reading url-http.el where again it is described
in a very confusing way:

(defvar url-http-attempt-keepalives t
  "Whether to use a single TCP connection multiple times in HTTP.
This is only useful when debugging the HTTP subsystem.  Setting to
nil will explicitly close the connection to the server after every
request.")

This is all the more irritating so many of the headers are set by
default using the variables url-vars.el. Why those things are at all
variables is a mystery to me. Then there are messages that appear in
the minibuffer and there is no way to disable them. In the end it is
much easier to do HTTP requests manually using make-network-process
then it is with the url library, it really isn't a good general
purpose HTTP component, there are too many completely opaque things
happening behind the scenes, like the hashmap of persistent tcp
connections that is maintained behind the scenes. Didn't anyone else
run into problems with it?

Cheers,
Jarosław Rzeszótko


2013/12/24 Stefan Monnier <monnier@iro.umontreal.ca>:
>> At the end of every HTTP request to be made with url-http.el and containing
>> a body, an unnecessary "\r\n" is appended, and additionally those two
>> characters are not used in the calculation of the Content-Length header.
>
> Looks like an old workaround.  Could someone get rid of this?
>
>
>         Stefan