Re: Non-greedy wildcard possible? (Long)

[Laurence Finston]

-------------------
> Laurence Finston <address@hidden>:
> >
> > If you don't signal "begin plain text" and "end plain text" somehow,
> > e.g., with balanced single quotes, as in your example, it would be
> > impossible for you to have plain text that contains tokens that
> > would be valid in parser rules.
> 
> Sure -- I do signal it. I just mean that there is no fixed such token;
> it all depends on context. (That doesn't mean that the grammar is/need
> be context sensitive, though.)

This seems to be a contradiction to me.  In a context-free grammar a token,
i.e., a terminal  symbol, is unique and unambiguous.  The Bison manual
describes a way of handling context dependencies, but it's fairly tricky. 
>From what you've said about the problem you're trying to solve, I have my
doubts about whether Bison is the right tool for the job.

> decide what tokens can be used as markup in a given context) is
> specified by the user, so I can't make any assumptions about it. (Atox
> is, in many ways, a parser generator.)

It sounds to me like you want Bison to generate a parser generator, but that's
not its intended purpose.  

> Yes, that's the problem. I cannot say that globally. I need the
> full power of the grammar to decide.

I don't understand how you're using the term "grammar".   To me,  "grammar"
(at least of the context-free variety) implies that there are syntactical
elements with a fixed value. 

> So, what I'm basically looking for is (as my original subject stated)
> a non-greedy wildcard mechanism in Bison (or, for that matter, some
> other parser generator -- Bison just seems readily available for the
> user). It's parallel to regexp for matching strings, for example; you

According to my understanding of regular expression matching algorithms, they
are by nature "greedy" in that they attempt to match the longest possible
string. 

I believe that a "wildcard" facility makes sense in a scanner, but not in a
parser.  Parsing tries to create a hierarchical structure from input; if
elements are read that don't fit into the structure, and error recovery
doesn't discard them or make them fit, it fails.  It's possible to change the
way  the scanner handles input at run-time, and to have it return different
tokens depending on context, but it's not possible to change the meanings of
tokens within the parser.

>From what I know of markup languages, there's no need for this kind of
hierarchical structure.  While the resulting documents may be structured
hierarchically, the input can be processed sequentially.   

Laurence