Re: Guile integration and UTF-8

[Paul Smith]

On Wed, 2013-09-25 at 14:56 +0200, Ludovic Courtès wrote:
> Eli Zaretskii <address@hidden> skribis:
> 
> >> From: Paul Smith <address@hidden>
> >> Date: Tue, 24 Sep 2013 22:31:29 -0400
> >> Cc: address@hidden, Lars Ljung <address@hidden>
> >> 
> >> I haven't made a formal definition, but I'm seriously leaning towards
> >> declaring that makefiles are UTF-8, always, and not supporting other
> >> encodings.
> >
> > Given that the difference is using scm_file_encoding instead of
> > scm_from_utf8_string, I don't see the reason why we should impose such
> > a restriction.  Using "-*- coding: -*-" cookies is quite commonplace
> > in the Free Software world, so it's not like Make will suddenly invent
> > a new protocol.  We can use UTF-8 as the default, in the absence of a
> > cookie, which I think will do what you want without unnecessarily
> > restricting those who might need a different encoding.
> 
> Agreed: coding cookie, and UTF-8 as the default.

I'm not making any decisions yet, and there will be opportunity for a
full discussion about it before I do.  However, I've been thinking about
this since before the Guile integration, and for reasons that have no
relationship to Guile.

I don't have time to get into this right now because I want to
concentrate on this release, and I need to organize my thinking first.

> > If you wonder when a non-UTF-8 encoding might be needed, then imagine
> > a Makefile which invokes some utility with a non-ASCII string argument
> > in a locale that isn't UTF-8.  Also, for arguments that come from the
> > Make command line (as in "make FOO='bar'") the string will come in the
> > locale's encoding, which will not always be UTF-8; in that case,
> > assuming UTF-8 is wrong, and we should use scm_c_eval_string as we do
> > now.  I'm sure there's any number of similar use cases out there.
> 
> Yes.

There are already serious restrictions on the types of encodings that
make will support.  For example, any encoding which does not reserve the
standard ASCII bytecodes for all characters which are special to make,
such as whitespace, newline, "$", "(", ")", "{", "}", ":", etc., and
which does not reserve a single byte '\0' as the EOS marker will simply
not work and cannot be made to work without VERY significant changes to
the code.