[Wesnoth-dev] New WML preprocessor now in CVS

[Guillaume Melquiond]

Hi,

Thanks to a 4-day vacation, I finally had time to finish the new WML
preprocessor I had mentioned a long time ago. Except for TRoW [1], all
the mainline campaigns seem to load fine. Consequently, I have
committed a first version, so that it gets broader testing.

First, concerning the performances. The good news: instead of
requiring the whole preprocessed output to be stored in memory, it can
now work on the fly. The preprocessing is done as soon as the
tokenizer (or whatever is reading the preprocessed output) has emptied
its input buffer. The bad news: it is slower due to the amount of
information inlined in the preprocessed stream in order to perform
full error location (load TRoW to see what I mean) and smart
textdomain tracking (see below). But anyway, once the preprocessed
stream has been cached, there is no performance difference: the binary
parser has been left unchanged.

The new preprocessor uses a recursive grammar, I already sent it to
the list and posted it on the forum. But I guess almost nobody
understood it, so I will just show the differences between the old and
the new preprocessor. The old one could not parse this sentence
{MACRO1 ({MACRO2 (A B)} C)} because the first right parentheses would
have matched the first left parenthesis. See bug #10995 for a real
scenario example.

The old preprocessor was also unable to handle nested conditional directives:
  #ifdef A
    #ifdef B
      xxx#else
      yyy#endif
  #endif
If A was not defined, it would have simply output "yyy" (whatever B
and without any error message) instead of outputting nothing as it
could be expected.

Macro definitions are the only part of the grammar that is not
recursive: you can't nest a #define...#enddef inside another one, and
you can't quote or comment #enddef.

Concerning #textdomain tracking, it is the only part that doesn't
follow my initial proposal, it is a bit clever now (and Yann won't
have to enhance wmlxgettext). Consider this example:

  #textdomain tda
  #define A B
    _"Something in tda" + {B}#enddef
  ...
  #textdomain tdb
  message={A _"Something in tdb"}

The translation of "Something in tda" won't be taken in the "tdb"
domain (where it will be included), it will be correctly searched in
the "tda" domain (where it is first defined). Moreover, the argument
"Something in tdb" of the macro won't be translated in "tda", but in
"tdb" (where it is defined).

Finally, text domains are no more WML-scoped, they last as long as no
new #textdomain directive has been encountered, and they get reset at
end of file and end of macro.

I will post part of this mail on the forum, so that campaign designers
are aware of the change. So if you have any question/remark that does
not concern the implementation but the grammar, it can/should be
posted on the forum.

Best regards,

Guillaume

[1] A small error with [time_area]. It doesn't prevent to play the
campaign. I'm still looking into why it fails.