Re: Locations suggest -- we're stupid

[Hans Aberg]

At 19:17 +0200 2002/01/09, Nikos Balkanas wrote:
>> Did you first, in the Flex source file main.c, check_options(), zip out
>>   if ( do_yylineno )
>>     /* This should really be "maintain_backup_tables = true" */
>>     reject_really_used = true;
>> recompile, and then do timings on a test with and without the %option
>yylineno?
>>
>> If so, what did the test show, on which computer?
>
>Nothing of the sort.

So then remove that line, and the redo the timing:

> Read from stdin into a buffer. yy_scanbuffer(). Fired
>up yylex(). Timed it (not profiled it) running through the patterns once.
>Then on EOF I yy_scanbuffer() again and throw yylex into a state that counts
>up new lines. Timed that too. This is faster than scanning yytext for '\n's
>using C. Showed consistent difference. By how much I don't remember.

Please give info about computer and timing differences. If it was a long
time ago, it may have changed, because the CPU's are now very fast
relatively to what they can read from the hard disk. To there is more time
over for examining each character.

> There
>is no point to it. No one is arguing to remove yylineno.

There is still a point getting to know the difference.

But with Unicode in place, one may hook up additional code converters
before the characters ends up with the Flex lexer, so the difference will
be even less then.

>I use overall time which is affected by I/O. One cannot do anything about
>disk I/O and the system calls for doing it. One can hope only to make fast
>software on top of that. Remember times are additive (Unless flex does I/0
>on a seperate thread which currently doesn't). Then again you have people
>that, realizing this, work from memory instead of a hard disk (after the
>first pass use
>yy_scanbuffer for subsequent passes). Furthermore, in development time when
>one debugs a particular input, many OS's (linux for example) will cache that
>input into memory (writing to disk, however, is the big hog, which cannot be
>cached).

The point is that on a modern computer system with paging, buffers, there
is no real way to ensure that those bottlenecks do not appear.

>Sorry, no excuse for slow code :-)

The main point is always that it does not matter if the code is slow if is
not executed frequently enough relative to the other stuff executed in the
program.

It may sound as a big over head, checking each character, but that is
already done in a number of times in buffers and code converters, etc. And
most overall time in a parser is typically not executed in the parser, but
in the actions.

> I see no tradeoff between structure and
>efficiency with flex.

This is already done when settling for a DFA: Some grammars create
exponential sized DFA's, and can thus not be implemented with Flex.

The tradeoff Flex does, is those grammars are relatively rare in common
practical use.

>I consider that the surest way to improve as a programmer is to consider
>one's
>own time inexpensive. Then one can turn some really nice (reusable) code. Of
>course that means putting extra time to it at off hours, initially.

That was the case in the earlyu days of compuers. This is not anymore the
case: Programmer time is always the most expensive part.

>However, this is of no interest to the list.

As a development strategy for Flex, it may, in view of the heated and long
discussions about lineno and locations.

>> But it was not exactly what is used in that IDE: Actually, it uses MacOS
>> file descriptors, and some other stuff I did not mention.
>
>So are you saying that the features proposed will not work with CodeWarrior?

In the program I made, one compiles only one file (.l or .y), which does
not include any other files. So there is only one file descriptor to keep
track of.

But if the files were stacked by includes, that would work.

> My argument based on the KISS principle

You have not yet explained what KISS is short for.

> is do
>not implement the features
>unless people really need it.

Well, that is the case of the line number feature.

> Those features won't be used by me or any
>of my friends. We are hooked on gcc and vim for now.

Clearly, if one is hooked onto a fixed, predetermined grammar that will not
change, than all kinds of tricks can be made to optimize that.

>Furthermore in the best case (no bugs - completely isolated code) there will
>be a delay in the parser generation (flex step is quite fast though).

The addition to the Flex compile times itself are clearly negligable.

> In
>addition it will make for a larger flex source, more difficult to maintain.

Not much.

>Akim seems to think that it might be unfeasible.

I think that Akim thinks about things that are more extensive than ever
proposed here.

> However, I am not the
>maintainer.

Neither am I: The maintainers mentioned (after my first email) that they
already have started on developing this issue, so I therefore provided some
more inputs.

> There are some things I wish I could see instead (egcs 2.91.66 -
>wrong pattern generation, I/O on a seperate thread).

Why do you not write a separate thread about what you want to see in Flex?

  Hans Aberg