Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > > From: Mark H Weaver <address@hidden>

 > > It doesn't matter how these raw bytes are encoded internally.  No
 > > matter what mechanism we use to accomplish it, propagating
 > > invalid byte sequences by default is bad security policy.
 > 
 > How can we be responsible for byte streams that originated outside?

By taking responsibility for them. ;-)

 > That's the responsibility of the source.  And if there is a consumer,
 > then it is their responsibility not to trip upon such bytes.

Not in a security context.  In a security context, you want defense in
depth: all separately developed components cooperate in covering up
each others' bugs by handling input carefully and refusing to transmit
broken output unless that is explicitly requested by the consumer (and
you trust it to know what it's doing when it says, "don't worry, I can
handle anything"!)

 > But how can you refuse to copy such bytes when you are just a pipe
 > that is expected not to change anything it wasn't toild to?

By signaling an error and terminating.  That's what a conformant
Unicode process does.

Note that the standard doesn't say you have to throw away state and
give up entirely.  It just says that if you do try to continue, you're
not conforming any more.

 > > The Unicode standard requires that all UTF-8 codecs refuse to
 > > accept, produce, or propagate invalid byte sequences, including
 > > the troublesome overlong encodings.

Again, what the Unicode standard says is that a codec that does any of
those things may not call itself conformant.  It doesn't say anything
about nonconformant codecs -- which include all non-Unicode codecs, of
course.  So we already know that Emacs is going to be nonconformant in
some use cases.  However, it should be aware of the fact when it is
not conforming to applicable standards.

 > So just passing a string unaltered through a Guile program would
 > change that string?

If it is a stream of bytes which is alleged to conform to some
standard but in fact does not, a program should (in general IMO, and
by the Unicode standard if it claims Unicode conformance) signal an
error and stop.  If the program wishes to pick up where it left off in
nonconformant mode and continue processing, that's fine (but it isn't
conformant any more).  In other words, "correcting" the string would
be a nonconformant behavior.

But this must be an explicit decision by the program (and it mustn't
do that if the user wants conformance).  Mark is absolutely correct
that the Guile system should not default to non-conformance, and I
don't think Emacs based on Guile should either, but that's up to the
Emacs community to decide, of course.