Re: GSoC: OOD detection

[Ramón García]

[OK, replying in public].

We could spend years in adding features to make. There is no limit to
one's imagination. I can imagine plugins for make syntax, plugins for
defining recipes and expression in whatever scripting language, of
course plugins for out of date determination, the ability of passing
arbitrary information from  a rule to the rule that makes its sources,
and vice-versa, and so on. The usual result of this featurits is
monsters like Microsoft Office or Mozilla. This is worse that it
seems, becuase added features cannot be then removed because of
backward compatiblity.

In order to make useful progress, we must try to find what is most
useful for a given effort. The restricted proposal by Paul is
completely general as far as out of date algorithms are concerted. The
implementation of persistency (mine), is certainly not general, but as
most of the time state files are only temporary things for
compilation, that are re-created from scratch by simply running make,
I can't see any usefulness in interoperating with other storage
schemes, apart from the beauty of generality.

For the case that you mention, ClearCase, it would be more useful
something that allows one to interpret compilation scripts for that
tool, perhaps translating these scripts to makefiles. Even with that
tool, I see little use for interoperating with compilation data: it is
not frequent for different users to share build directories. Doing so
would be a nightmare: developer Helen makes an change to source, it
does not compile the first time, and while she fixes it, developer
Mark who is debugging a corner case is interrupted, because the
program does not build, so it is not posible to run or debug it.

Returning to the main topic, I cannot see enough need for a feature of
user-defined storage that justifies the increased complexity.

Talking about details, the proposed (by Paul, not by me) out of date
detection would not require the creation of one process for every out
of date evaluation. It would be an expression evaluation. Only of the
user explicitly requests a $shell function, that would be a command.
And this project includes adding the builtin functions neccessary for
the common cases (md5sums for instance). With a builtin function, the
evaluation for common cases would be as simple as
.OUT_OF_DATE="md5sum", or perhaps something like
.OUT_OF_DATE="$(md5sum $^)".


> 1. It seems to assume/require file-based statefulness, which is
> harder to share between users and across systems.
>
> 2. It implies to me that some OOD mechanisms will require a fork/exec
> (or worse, CreateProcess) for each OOD determination. This would be
> orders of magnitude slower than a stat system call.
>
> 3. It sort-of requires makefile changes. As you noted, the
> .OUT_OF_DATE var could be supplied in the environment or command
> line, or in an include file requiring only minor changes to the main
> makefile. But in order to do anything target-specific or using
> complex expressions involving $(shell) or $(eval) or Guile, my gut
> feeling is that it would be pretty ungainly doing anything other than
> putting it in the makefile. The command line gets into quoting
> issues, the environment means that every command make runs
> (compilers, linkers, etc) is going to see that EV too (thus stealing
> from the ARG_MAX pool), etc.
>
> Turning to use cases: I have serious hopes of implementing a system
> in which the data required to do OOD determination is held in a
> relational database available only via SQL queries over a socket. Can
> you tell me how I would do that with your proposed .OUT_OF_DATE and
> statefulness proposals? I have no particular vested interest in a
> "complete plugin system", I just want something that will solve the
> problem generically and extensibly. The plugin model does appear to
> do so with the advantage of staying in one process space but I'll be
> happy to abandon it if you can show me that your proposal would work as well.
>
> Let me finish by repeating a point I made previously: to my mind the
> hard work is finding all the places where stat() is used for OOD and
> bringing them through one place. Once that's done, I could always
> submit my own patch to do a plugin system and make a case for it
> independently. So I very much appreciate your willingness to do the
> main work here and I ask only that the final design not preclude
> further extensibility. Or, if you can show me how the system you
> propose would solve my problem or vice versa, so much the better.
>
> Thanks,
> David Boyce