Control job launch via external program

[Jaak Ristioja]

Hello,

GNU make allows the user to control the number of simultaneous running 
jobs using by the -j, --jobs option, and to prevent new jobs from 
starting based on the load average by using the -l, --load-average 
option. These options currently provide very limited means for users to 
prevent the system from running out resources, most notably available 
memory[1].

Adding new --some-resource options in addition to --load-average does 
not seem to be very reasonable, as it might lead to in endless 
discussions about which resources to account for and how exactly to do 
that. For flexibility, I believe such decisions are best left to the user.

Would it please be possible to enhance GNU make with a command-line 
option taking a string argument to specify a name or path to an external 
program to further control starting of jobs or recipes?

Here are some of my raw ideas on this.

One could for example add an option similar to -l which would take the 
name or path of an executable to executed whenever the load average 
would be checked with -l. Whether or not the respective job should be 
started could depend on the exit status of that program. Any auxiliary 
information to the program could be provided via command-line arguments 
or (prefixed) environment variables.

It were possible to extend the -l option to parse a non-number argument 
as a name or path to such a program, but the long option name 
"--load-average" would in my opinion not best suit such use. According 
to the make(1) manpage the limit set using -l is only applied when 
"there are others jobs running" (should read "other"?), meaning it can 
not be used to delay execution of all jobs as in case of only one job it 
will not be subject to this limit.

It would perhaps also be possible extend the -j option to take an 
executable (e.g. `-j nproc`) so that new jobs would not be started if we 
are already running more jobs than a number returned by the external 
program. Note that it is also possible for the external program to not 
exit immediately, but to wait on resources to become available. I must 
say I haven't yet figured out a set of good semantics for this one.

Another way might be to provide a launcher executable (e.g. via 
--launcher or similar) to be used in place of the shell to execute the 
Makefile recipes, so that the executable could fully control each such 
recipe invocation. Whenever make would execute a recipe, it would 
instead pass the recipe (or parts thereof) and other relevant 
information (including SHELL and .SHELLFLAGS) to the launcher, which 
will then run the recipe. If the launcher detects insufficient resources 
it can delay actual execution of the recipe until sufficient resources 
become available. The launcher could then be used in addition to any -j 
and -l options (with or without arguments). The launcher could also be 
used for other things besides job control, such as debugging or 
measuring resource usage. This idea is partly borrowed from CMake which 
has RULE_LAUNCH_(COMPILE|LINK|CUSTOM) properties to allow, for example, 
for easy use of ccache.

Not being too familiar with the intricacies of GNU make, my take on this 
might be naive, but I'm sure other users would also appreciate such a 
feature. Thanks! :)


Best regards,
Jaak Ristioja


[1]: https://stackoverflow.com/questions/11639524/make-j-ram-limits