Re: Google summer of code for GNU make: introducing myself.

[David Boyce]

At 08:19 PM 4/12/2007, Ramón García wrote:
> The problem that this project attempts to solve is to allow the user
> to define how it is determined if a target is out of date. By
> providing this facility and providing persistent variables, it should
> be possible to write simple Makefiles that contain dependencies on
> CFLAGS, or on the rule of the command. Following Paul's suggestion,
> the approach that is most likely to be followed, is to have a special
> variable name, .OUT_OF_DATE, that will contain an expression whose
> evaluation will determine if the target must be rebuilt. In addition
> we will have to choose among several ways to store persistent
> variables.

Ramon,

Given the quick succession of announcement, design suggestion, and 
prototype this is looking more like a Spring of Code :-). I'm not 
sure if it's not already too late to make alternate suggestions, but 
I'd prefer a design which does not require modifying the makefile. 
After all, the method of determining out-of-dateness (hereafter OOD) 
is orthogonal from what the makefile encodes, which is simply (a) 
what files depend on what other files and (b) what commands to run 
when an OOD situation is discovered.

I have in the past suggested a plugin style design, which to the best 
of my understanding would require two pieces of work:

1. Refactor the GNU make code so that all OOD determination is done 
in a single function (currently there are stats and date comparisons 
spread out all over). Of course this part would presumably have to be 
done for any implementation, and of course there would be a built-in 
function providing the traditional stat/compare behavior.

2. Then, at startup, make could look (by some to-be-determined 
mechanism[*]) for a shared library/DLL providing this function. If 
found, it loads that shared lib and  registers said function as the 
OOD determinator. If not, it registers the builtin.

I don't think the second part would be hard; it's a simple matter of 
using the common dlopen/dlsym functions on Unixlike systems and 
equivalent LoadLibrary() etc on Windows. I believe there are 
abstraction layers which cover up platform differences here. I'm 
pretty sure Apache APR has one and there may well be one within the 
GNU universe too.

So the way I see it is: (1) is the hard part and has to be done no 
matter what. (2) is relatively easy and would provide an open 
environment for people to provide their own OOD algorithm. You could 
implement your .OUT_OF_DATE design on top of it as a proof of concept 
while leaving the door open for alternate/contributed solutions.

Issues I can think of:

-> There may be an older platform or two which don't support dlopen 
or an equivalent. But these are going to be few and fading away: 
dlopen is in POSIX and is implemented on all viable modern Unixlike 
systems. And I don't know what the official policy is but I don't 
think it's a requirement that all new functionality be supported on 
all old platforms. Most SW providers, free and otherwise, have no 
problem saying that obsolete platforms may be supported but will not 
necessarily participate in all new features.

-> Writing a shared library plugin is harder, and requires a 
different skill set, than putting a shell command in the .OUT_OF_DATE 
variable (if I understand your proposal correctly). This is quite 
true, especially considering that shell commands are inherently more portable.

So why not do the plugin thing, with the builtin function 
implementing the strategy you propose: looking for a .OUT_OF_DATE and 
dropping back to timestamp? That seems like a win all the way around 
to me. At the very least, if there's nothing wrong with the plugin 
idea we could keep it on the roadmap and try to design the 
.OUT_OF_DATE work in a way that's compatible with it.

I don't mean to make a big deal out of this. If your contribution 
ends up solving part (1) then there would be nothing stopping me or 
someone else from submitting a patch to do the rest and make our own 
case for it separately. So any work that refactors the OOD code into 
a single place will be a great thing. But at the same time nobody has 
yet shot down the plugin model and I'd prefer to see it debated 
before a final plan is settled on.

Thanks for doing this work,
David Boyce

[*] The mechanism could be to look for a particular environment 
variable pointing to a shared object or similar. Better yet, a make 
variable which could be derived from the env or a property of the 
makefile. The makefile should be fully parsed before the first OOD 
determination is made, right?

PS I'm out of town and off line for a week starting tomorrow.