symbol_type::token () removed?

[Wolfgang Thaller]

Commit 8c87a623 says:

> c++: get rid of symbol_type::token ()
> It is not used.  And its implementation was wrong when api.token.raw
> was defined, as it was still mapping to the external token numbers,
> instead of the internal ones.  Besides it was provided only when
> api.token.constructor is defined, yet always declared.
>
> * data/skeletons/c++.m4 (by_type::token): Remove, useless.

Well, I disagree. It's not useless, I'm using it, and I think my use
case is legitimate.

Low priority, I have an easy workaround.


My parser uses the C++ parser template with

%define api.token.constructor
%define api.value.type variant

I've got a hand-written lexer which returns symbol_type.
And because I'm a good boy, I have unit tests for my lexer. And in my
unit tests, I call my lexer on small bits of input, and then I need to
inspect the resulting symbol_type and make sure the right kind of token
is returned:

So I have things like:

        Lexer lex("12345");
        symbol_type t = lex.nextToken();
        assert(t.token() == token::INTLIT);
        assert(t.value.as<int>() == 12345);


... which suddenly stopped working.

Fortunately, I can instead do...

        assert(t.type_get() == by_type(token::INTLIT).type_get());

... but that's a bit ugly.

In general, I think symbol_type should always provide enough public API
in order to allow inspecting it. In my case, I only need it for
implementing tests, but it general it might be useful when writing the
lexer, as well.

So I request to reinstate symbol_type::token().



Cheers,

Wolfgang



P.S.: Keep up the good work, bison is my favorite way of parsing things
using C++....