Re: url-retrieve-synchronously results differ from curl

[Sean Allred]

Artur discovered something this morning that I wanted to share here as he works 
on a proper bug report. (We’re both especially busy today, unfortunately.) 
Perhaps someone can shed light on this behavior.

Binding `url-request-method` to `”POST”` rather than `”post”` seems to fix the 
issues en masse. This has not undergone proper testing, but it definitely seems 
to work [1]. I apologize that I don’t have an example that works out-of-the-box 
– I really have a very limited experience with web stuff – but using “POST” 
rather than “post” makes a significant difference.

The question we both have at this point is *why* – *why* does “post” ‘partly’ 
work? *Why* does “POST” work fully? And frankly, in my personal opinion, *why* 
aren’t these methods taken in as symbols in the first place?

Either Artur or I will be filing a proper bug report for url.el on this issue 
as time allows (assuming of course that someone doesn’t beat both of us to it).

(Unless this thread can count as a bug-report?)

All the best,
Sean Allred

[1]: https://gist.github.com/90be70f12cd097be1247 
<https://gist.github.com/90be70f12cd097be1247>, reproduced below

(defconst tmp:access-token
  ;; Needed to post answers.  Considered a secret.  If you would like
  ;; a key, use `sx-authenticate' from sx.el (available from MELPA)
  ;; and look in ~/.emacs.d/.sx/auth.el
  "YOUR ACCESS TOKEN HERE")

(defconst tmp:key
  ;; not considered a secret
  "0TE6s1tveCpP9K5r5JNDNQ((")

(defun tmp:api-bug (use-curl access-token key)
  "Post a test answer to the formatting sandbox."
  (let ((random-body-1 (md5 (current-time-string)))
        (random-body-2 (md5 (md5 (current-time-string))))
        (method "https://api.stackexchange.com/2.2/questions/3122/answers/add";)
        (args
         (mapconcat
          #'identity
          `(,(format "access_token=%s"
                     (replace-regexp-in-string
                      "%" "%%" (url-hexify-string access-token)))
            ,(format "key=%s"
                     (replace-regexp-in-string
                      "%" "%%" (url-hexify-string key)))
            "site=meta"
            "pagesize=100"
            
"filter=%%21GoYr1we0U5inG5G7wBg4JBGpbgX%%29C7LDqpy-%%2AbfwPOujOr4SR4W%%29bLNSyYUpQDdTwTj.XChTFB0gfLaAJq0hv"
            "body=this-is-an-answer-test-for-sx.el--%s")
          "&")))

    (if use-curl
        (shell-command-to-string
         (format
          "curl --silent -X POST --data %S %s | gunzip"
          (format args random-body-2)
          method))
      (let ((url-automatic-caching t)
            (url-inhibit-uncompression t)
            (url-request-data (format args random-body-1))
            ;; emacs-devel: note vvvv
            (url-request-method "POST")
            (url-request-extra-headers
             '(("Content-Type" . "application/x-www-form-urlencoded"))))
        (with-current-buffer
            (url-retrieve-synchronously method)
          (goto-char (point-min))
          (search-forward "\n\n")
          (delete-region (point-min) (point))
          (buffer-string))))))

(require 'json)

(json-read-from-string (tmp:api-bug t tmp:access-token tmp:key))

((total . 1)                            ; this data structure used curl
 (page_size . 100)
 (page . 1)
 (quota_remaining . 9979)
 (quota_max . 10000)
 (has_more . :json-false)
 (items .
        [((link . "http://meta.stackexchange.com/questions/3122//247295#247295";)
          (body_markdown . 
"this-is-an-answer-test-for-sx.el--e3eeb6228ed9c2c58e5385b73493f0f0")
          (share_link . "http://meta.stackexchange.com/a/247295/188148";)
          (answer_id . 247295)
          (creation_date . 1421684370)
          (last_activity_date . 1421684370)
          (score . 0)
          (upvoted . :json-false)
          (downvoted . :json-false)
          (owner
           (display_name . "Sean Allred")
           (reputation . 160)))]))

(json-read-from-string (tmp:api-bug nil tmp:access-token tmp:key))

((total . 1)                            ; this data structure used "POST"
 (page_size . 100)
 (page . 1)
 (quota_remaining . 9999)
 (quota_max . 10000)
 (has_more . :json-false)
 (items .
        [((link . "http://meta.stackexchange.com/questions/3122//247396#247396";)
          (body_markdown . 
"this-is-an-answer-test-for-sx.el--5b447b87e7078ed0fd34a3169ee84319")
          (share_link . "http://meta.stackexchange.com/a/247396/188148";)
          (answer_id . 247396)
          (creation_date . 1421847252)
          (last_activity_date . 1421847252)
          (score . 0)
          (upvoted . :json-false)
          (downvoted . :json-false)
          (owner
           (display_name . "Sean Allred")
           (reputation . 160)))]))


((total . 1)                            ; this data structure used "post"
 (page_size . 100)
 (page . 1)
 (quota_remaining . 9977)
 (quota_max . 10000)
 (has_more . :json-false)
 (items .
        [((link . "http://meta.stackexchange.com/questions/3122//247297#247297";)
          (share_link . "http://meta.stackexchange.com/a/247297/188148";)
          (answer_id . 247297)
          (creation_date . 1421684555)
          (last_activity_date . 1421684555)
          (score . 0)
          (owner
           (display_name . "Sean Allred")
           (reputation . 160)))]))