mid-rule actions and lexical ties-in

[Guillaume Rousse]

Unfortunatly for me, I have binary blobs embedded in text streams to
read (some people think that's the only way to exchange float numbers
without precision loss). They have no delimiters, preventing me from
using scanner-based decision to switch scanning mode, as commonly done
for scanning strings, for instance. They can however only happen in
given context, established by grammar, and with fixed length
(sizeof(double) actually, meaning 8 bytes on 32 and 64 platforms).

For instance, a binary float arrays is something as:
begin
[1%fb1]
[1
<1>
0
BINARY_BLOB;
1]
end

>From bison documentation, it appears I may use mid-rules actions to
trigger arbitrary actions, including scanner state. I could not invoke
yy_push_state() directly, however, as it is just a flex macro, but I
could use an user-defined function. I tried the following combination:

parser.y:

float_binary_1D_lines:
      /* empty */
    | float_binary_1D_line
    | float_binary_1D_lines float_binary_1D_line
    ;

float_binary_1D_line:
                        { set_binary_context(); }
    VALUE_FLOAT_BINARY
    SEMICOLON EOL       { _float_1D_value_item($1); }
    ;


scanner.l:

/* default chunk size is too large,
   and line-based reading doesn't work */
#define YY_READ_BUF_SIZE 1

<BINARY>.{8}   {
    yylval.f = ((double*) yytext)[0];
    yy_pop_state();
    return VALUE_FLOAT_BINARY;
}

%%
void set_binary_context(void) {
    yy_push_state(BINARY);
}

However, it doesn't work:

Entering state 160
Reducing stack by rule 98 (line 384):
-> $$ = nterm float_binary_1D_lines ()
Stack now 0 1 8 16 21 25 66 107 160
Entering state 199
Reading a token: --(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting default rule ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting default rule ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting default rule ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting default rule ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting default rule ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting default rule ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting default rule ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting default rule ("")
--accepting rule at line 78 (";")
Next token is token ";" ()
Reducing stack by rule 101 (line 391):
setting binary mode
-> $$ = nterm @7 ()
Stack now 0 1 8 16 21 25 66 107 160 199
Entering state 201
Next token is token ";" ()
syntax error, unexpected ;, expecting VALUE_FLOAT_BINARY at line 8

Basically, it seems scanner state change occurs after default rule catch
and discards individual bytes. I tried moving the mid-rule action to a
non-terminal symbol as explained in documentation, so as to change its
execution order without succcess. Using a global variable, as original
exemple in 'lexical tie-ins' uses, instead of a variable call, doesn't
change anything.

I tried an alternative strategy, based on explicitely matching .,
storing it elsewhere, and pushing it back when needed, but it doesn't
work either:

scanner.l:

.       { value_buffer[value_index++] = *yytext; }

void push_binary_value_back(void) {
    int i;
    fprintf(stderr, "pushing back 8 last bytes\n");
    for (i = 7; i <= 0; i--)
        unput(value_buffer[value_index - i]);

    yy_push_state(BINARY);
}

parser.y:

float_binary_1D_line:
                                     { push_binary_value_back(); }
    VALUE_FLOAT_BINARY
    SEMICOLON EOL { _float_1D_value_item($2); }
    ;

Entering state 160
Reducing stack by rule 98 (line 385):
-> $$ = nterm float_binary_1D_lines ()
Stack now 0 1 8 16 21 25 66 107 160
Entering state 199
Reading a token: --(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting rule at line 155 ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting rule at line 155 ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting rule at line 155 ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting rule at line 155 ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting rule at line 155 ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting rule at line 155 ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting rule at line 155 ("")
--(end of buffer or a NUL)
--(end of buffer or a NUL)
--accepting rule at line 155 ("")
--accepting rule at line 64 (";")
Next token is token ";" ()
Reducing stack by rule 101 (line 392):
pushing back 8 last bytes
-> $$ = nterm @7 ()
Stack now 0 1 8 16 21 25 66 107 160 199
Entering state 201
Next token is token ";" ()
syntax error, unexpected ;, expecting VALUE_FLOAT_BINARY at line 8

Am i missing something there ?

-- 
Guillaume Rousse
Projet Estime, INRIA
Domaine de Voluceau
Rocquencourt - B.P. 105
78153 Le Chesnay Cedex - France