[Lynx-dev] Lynx mangles some UTF-8 chars on POST (KMM184778069V18779L0KM)

[james]

On Sun, Feb 26, 2012 at 07:58:17PM -0800, Yahoo! Mail wrote:
> Hello, 
> 
> Thank you for contacting Yahoo! Mail Customer Care.
> 
> I'm not really clear on the issue that you are experiencing.
> 
> To help us get a better understanding of the issue, please reply with 
> the following details:<SNIP!>
> 
> Regards,
> 
> Wendy
> 
> Yahoo! Mail Customer Care
> 
> 92482568 
> CA Mail Tech E2Y
>

When I said the message was deleted on POST, I was using technical 
jargon referring to the HTML/HTTP from submission. Briefly the steps to 
reproduce were:
1. Create New message
2. Fill out 'To', 'Subject' and body with a *specific* message.
3. Select submit/send
4. Observe message body is missing in sent mail folder.

At the time of my original message, I had verified that my ISP was 
likely not tampering with the message by using a second ISP that uses a 
separate routing path.

I have since done further testing in an attempt to rule out my 
web-browser as the problem. It turns out that my web-browser was 
submitting invalid UTF-8 data during POST. Yahoo! mail blanking the 
invalid field was likely correct, if cryptic behaviour.

I have attached sample text and a Lynx.trace log that can be shared 
publicly. I have cc'd the lynx-dev mailing list. I have not yet tested 
with the latest development version of lynx.

The UTF-8 character string in question was: E2 80 99
On the wire (line 139 of the Lynx.trace) this gets sent as:
%C3%A2%80%99, an invalid UTF-8 sequence. The correct string would be:
%E2%80%99
Other examples in sample text:
EF BF BD -> %C3%AF%C2%BF%C2%BD
Note: It appears I copy&pasted the substitution character. The resulting 
sequence is valid, but appears to be "dumb" per-byte code-page 
conversion.

Regards,

James Phillips

 
<SNIP!> 
> 3.  Provide your computer system information.
> ---------------------------------------------
>       Operating System: Linux (MIPS 64)
> 
>       Type of Browser: Lynx 2.8.7dev.9 (27 Apr 2008) (development 
> version)
> 
-- 
OpenPGP Public Key: http://phillipsjk.ca/signature0611.txt

sample.txt
Description: Text document

declassified.Lynx.trace.txt.gz
Description: Binary data

signature.asc
Description: Digital signature