Unable to `url-retrieve' over HTTPS via proxy at work (but curl works fine)

[Jay Cotton]

Hi,

`url-retrieve’ gets status=400 when I try to retrieve an HTTPS URL. It works 
fine for HTTP.

My proxy is configured with an environment variable, and debugging the process 
I was able to confirm that it is finding the correct proxy.

I tested the same URL with curl, and that worked fine.

I turned on all logging by setting url-debug=t, and I ran curl in verbose mode 
and compared the output from each. It seems like curl is using the CONNECT 
method to create a tunnel, then passing the GET request to the non-proxy host 
through the tunnel, whereas `url-retrieve’ is sending the GET to the proxy (and 
getting status=400).

Does anyone know how I can get the emacs url library to behave like curl?

I’m pasting the debugging output below. First the output from *URL-DEBUG*, then 
the output from curl.

Thanks,
Jay

******* From emacs (I turned on all logging to the *URL-DEBUG* buffer):

http -> Contacting host: webproxy.my-domain.com:8080
http -> Marking connection as busy: webproxy.my-domain.com:8080 #<process 
webproxy.my-domain.com>
http -> Request is: 
GET 
https://api.forecast.io/forecast/3dac162e326fbefa4a448e33e0df0c45/32,79?lang=en&units=si
 HTTP/1.1
MIME-Version: 1.0
Connection: close
Extension: Security/Digest Security/SSL
Host: api.forecast.io
Accept-encoding: gzip
Accept: */*
User-Agent: URL/Emacs


http -> Calling after change function 
`url-http-wait-for-headers-change-function' for `#<process 
webproxy.my-domain.com>'
http -> url-http-wait-for-headers-change-function ( *http 
webproxy.my-domain.com:8080*-529274)
http -> Saw end of headers... ( *http webproxy.my-domain.com:8080*-529274)
http -> url-http-parse-response called in ( *http 
webproxy.my-domain.com:8080*-529274)
http -> Got a content-length, being smart about document end.
http -> Calling initial content-length for extra data at end of headers
http -> Marking connection as free: api.forecast.io:443 #<process 
webproxy.my-domain.com>
http -> url-http-parse-headers called in ( *http 
webproxy.my-domain.com:8080*-529274)
http -> url-http-parse-response called in ( *http 
webproxy.my-domain.com:8080*-529274)
http -> Parsed HTTP headers: class=4 status=400
http -> Finished parsing HTTP headers: t
http -> Marking connection as free: api.forecast.io:443 #<process 
webproxy.my-domain.com>
http -> Activating callback in buffer ( *http 
webproxy.my-domain.com:8080*-529274)
http -> Spinning waiting for headers...


******* From curl:

[1] 39840
14:19:32 ~ $ * Adding handle: conn: 0x7fb65a803a00
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fb65a803a00) send_pipe: 1, recv_pipe: 0
* About to connect() to proxy webproxy.my-domain.com port 8080 (#0)
*   Trying 172.30.50.205...
* Connected to webproxy.my-domain.com (172.30.50.205) port 8080 (#0)
* Establish HTTP proxy tunnel to api.forecast.io:443
> CONNECT api.forecast.io:443 HTTP/1.1
> Host: api.forecast.io:443
> User-Agent: curl/7.30.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied OK to CONNECT request
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
* Server certificate: darksky.net
* Server certificate: DigiCert SHA2 Secure Server CA
* Server certificate: DigiCert Global Root CA
> GET /forecast/3dac162e326fbefa4a448e33e0df0c45/32,79?lang=en HTTP/1.1
> User-Agent: curl/7.30.0
> Host: api.forecast.io
> Accept: */*
> 
< HTTP/1.1 200 OK
* Server nginx/1.1.19 is not blacklisted
< Server: nginx/1.1.19
< Date: Fri, 28 Aug 2015 18:19:32 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 18565
< Connection: close
< X-Powered-By: Express
< X-Hostname: june
< X-Forecast-API-Calls: 6
< Cache-Control: max-age=2428
< Expires: Fri, 28 Aug 2015 19:00:00 +0000
< ETag: "-2033433468"
< X-Response-Time: 140ms
< 
{"latitude":32,"longitude":79,"timezone":"Asia/Kashgar","offset":6,"currently":{"ti