argument collection vs. tokens

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This is an extreme corner case, so I'm asking here to get any feedback
before I try changing the behavior in 1.4.x.

When you use changecom or changequote to specify a start sequence that
begins with '(', macro.c currently uses peek_input() to see that the next
character is '(', then calls next_token() to discard the entire token.
However, next_token() gobbles the entire comment or quote, leading to
weird output (at least we can recover; when I tried this on Solaris, the
input engine gets stuck and claims that the entire rest of the file is an
unterminated quote or comment):

$ m4
define(echo,`$#:$@')changecom(`(',`)')dnl
echo(1,2)(3,4),5,6)
3:(3,4),5,6

My thoughts is that only input.c should be peeking at characters; macro.c
should only be concerned with tokens.  It would be better if we had a
function in input.c called peek_token() which determined if the next token
would be TOKEN_SIMPLE with contents '(' or not, so that the case above
would become:

$ m4
define(echo,`$#:$@')changecom(`(',`)')dnl
echo(1,2)(3,4),5,6)
0:(1,2)(3,4),5,6

Setting the start character to a single ( disables argument collection.
But it should be possible to use a multi-character quote or comment start,
and tell argument collection apart from quotes.  Again, my proposed idea
of peek_token() would cause this behavior:

$ cat <<\EOF | m4
changequote(`(-', `-)')dnl
define(a,A)
define(-b,B-)
a b
EOF

defineb,B
A b

- --
Life is short - so eat dessert first!

Eric Blake             address@hidden
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFE0KRN84KuGfSFAYARAtZiAJ0a2H0EDESN4DqKsDY1fQGd/vodOwCgu0FJ
ivFsxzpcFuFX1EnGZgb2M5I=
=WVEx
-----END PGP SIGNATURE-----