Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

Eli Zaretskii <address@hidden> writes:

>> From: Thomas Fitzsimmons <address@hidden>
>> Date: Mon, 14 Mar 2016 10:58:44 -0400
>> Cc: Stefan Monnier <address@hidden>, address@hidden
>> 
>> My plan is to replicate that without the soap-client patch, and try to
>> fix it a different way, in Debbugs.  From that thread, I think this will
>> show it:
>> 
>> (async-get (async-start
>>             `(lambda ()
>>                (load ,(locate-library "debbugs"))
>>                (debbugs-get-status 22285))))
>> 
>> I'll work on this hopefully this evening unless you beat me to it.
>
> Thanks.  But it would be good to discuss what you think should be done
> in debbugs.el, before you actually do that.  The important part is to
> figure out how can debbugs.el know whether a byte stream is a text
> string.

With the soap-client patch reverted, this debbugs.el patch fixes the
above test case, producing a correct multibyte string for originator.

This is just one example, obviously.  The new function should be applied
to all values returned by the Debbugs server that may contain multibyte
UTF-8 characters.  I'll leave this part to Michael.

Also, there's still something not quite right about async, for strings
that don't contain extended characters:

(multibyte-string-p (cdr (assq 'severity (car (async-get (async-start
            `(lambda ()
               (load ,(locate-library "debbugs"))
               (debbugs-get-status 22285))))))))
=> nil

(multibyte-string-p (cdr (assq 'severity (car (debbugs-get-status 22285)))))
=> t

but this looks like a side effect of async.

I'll wait to revert the soap-client patch on master and emacs-25 until
Debbugs has a new release to fix this, to avoid temporarily re-breaking
Debbugs.  (However, I'd really like to get this done before Emacs 25.1
so that it doesn't go out with an incompatible soap-client API.)

Thomas

diff --git a/packages/debbugs/debbugs.el b/packages/debbugs/debbugs.el
index f145280..e4c9667 100644
--- a/packages/debbugs/debbugs.el
+++ b/packages/debbugs/debbugs.el
@@ -264,6 +264,27 @@ (defun debbugs-newest-bugs (amount)
   "Return the list of bug numbers, according to AMOUNT (a number) latest bugs."
   (sort (car (soap-invoke debbugs-wsdl debbugs-port "newest_bugs" amount)) '<))
 
+(defun debbugs-convert-soap-value-to-string (string-value)
+  "If STRING-VALUE is unibyte, decode its contents as a UTF-8 string.
+If STRING-VALUE is a multibyte string, then `soap-client'
+received an xsd:string for this value, and will have decoded it
+already.
+
+If STRING-VALUE is a unibyte string, then `soap-client' received
+an xsd:base64Binary, and ran `base64-decode-string' on it to
+produce a unibyte string of bytes.
+
+For some reason, the Debbugs server code base64-encodes strings
+that contain UTF-8 characters, and returns them as
+xsd:base64Binary, instead of just returning them as xsd:string.
+Therefore, when STRING-VALUE is a unibyte string, we assume its
+bytes represent a UTF-8 string and decode them accordingly."
+  (if (stringp string-value)
+      (if (not (multibyte-string-p string-value))
+         (decode-coding-string string-value 'utf-8)
+       string-value)
+    (error "Invalid string value")))
+
 (defun debbugs-get-status (&rest bug-numbers)
   "Return a list of status entries for the bugs identified by BUG-NUMBERS.
 
@@ -421,6 +442,11 @@ (defun debbugs-get-status (&rest bug-numbers)
            (when (stringp (cdr y))
              (setcdr y (mapcar
                         'string-to-number (split-string (cdr y) " " t)))))
+         ;; "originator" may be an xsd:base64Binary value containing
+         ;; a UTF-8-encoded string.
+         (dolist (attribute '(originator))
+           (setq y (assoc attribute (cdr (assoc 'value x))))
+           (setcdr y (debbugs-convert-soap-value-to-string (cdr y))))
          ;; "package" is a string, containing comma separated
          ;; package names.  "keywords" and "tags" are strings,
          ;; containing blank separated package names.