Re: [Grammatica-users] Easier lexer generation.

[Matti Katila]

On Tue, 28 Jun 2005, Per Cederberg wrote:
> Well, the Grammatica lexer is essentially identical to most other
> tools. It provides support for defining tokens via regexps (and
> plain strings). If you want to use EBNF, you can just write those
> "tokens" as productions instead.

This would make the division to lexer and parser much harder. For
example python lexer which marks_ indent and dedent is easy to do after
lexing - we may call that post-lexer. Post-lexing is much easier to do
after long literals_ and comments are regocnized already.

_marks =: http://docs.python.org/ref/indentation.html
_literals =: http://docs.python.org/ref/strings.html

Of course, I don't know yet since I haven't tried it out, I could create
foolexer.grammar and fooparser.grammar for grammar foo and use some sort
of wrapping with grammatica to use lexer.grammar as lexer where I want to
use ebnf in lexer grammar.

> The general problem with grammars where tokens are specified in
> EBNF is efficiency and ambiguities.

Efficiency sounds as premature-optimizing which shall considered as no-no :)

> That is not a problem in a language specification document where
> readability is the highest concern.

I think readability is high concern in every case :) Well, I think I know
what you mean.


   -Matti