New feature $(flock ...) - I'll try to implement myself and provide patch

[Zoltan]

I'd like to suggest and implement a new feature in make to control
parallelism.  I'm basing my suggestion on an old post from Paul Smith:

http://lists.gnu.org/archive/html/bug-make/2012-07/msg00007.html

I've implemented this in my own projects, though without the "uniqueness" of
different lock-file names for multiple 'make'-s running in different sessions
as I don't need that uniqueness feature at the moment.

In the course of implementing the shell-based flock in a makefile, here's my
rule, slightly sanitized for clarity:

# Serialize the install targets:
%/install:
        flock $(SOMEDIR)/install_lock $(MAKE) -C $(@D) $(notdir $(@))

This works as expected, and I like to credit Paul for his suggestion.

Now, after observing this behavior at build time, it occurred to me that at
least the theoretical implementation of an $(flock ...) function _inside_ make
itself would provide a major additional benefit that it could release the
job-slot while the $(MAKE) currently under execution was waiting for the lock,
thus not holding an active job-slot hostage while it waited for the lock,
unlike the shell-based flock solution.

In watching my current (shell-based) implementation running with a -j8 or even
a -j16, I have seen the make reduced to an effective -j1 when my jobs all hit
the same flock.  This is not ideal, and completely defeats the purpose of
parallel builds for that particular target, since it consumes all the job
slots but only one can be executing.  This is not a "fault" of make, but make
could offer a solution with an internal $(flock ...).

So the advantage of an internal $(flock ...) implementation are several:

1)  release job slot while waiting on lock, thus keeping the number of slots
available for runnable jobs at the user-specified "-j X" level

2)  one $(MAKE) (in a session) does not share locks with other $(MAKE)'s in a
different session, thus solving an issue that Paul himself pointed out right
after his post that I reference above.

>From an implementation standpoint, the job-slot count manipulation is not a
new concept, as sub-makes already exempt themselves and that mechanism should
be usable (I'm just guessing here, but it seems logical) for a similar
exemption, based upon state as follows:  release job-slot while waiting for
lock, once lock has been obtained transition to "runnable" job-state as would
normally happen (so await available job-slot and continue execution as normal
while holding lock), do nothing to job-slot when lock is released.

So there are several design choices that occur to me, revolving around
formatting of the command itself:

$(flock lock-name) rest-of-target-rule

This would make my code snippet above look like

%/install:
        $(flock install_lock) $(MAKE) -C $(@D) $(notdir $(@))

another option is:

$(flock lock-name command-to-run-while-holding-lock)

yielding:

%/install:
        $(flock install_lock $(MAKE) -C $(@D) $(notdir $(@)))

and there may be other options.  I'm open to suggestions, though I am leaning
towards the first [$(flock lock-name) rest-of target-rule)] because it might
be easier to parse, and we could treat the presence of an $(flock ...)
function anywhere within a particular instance of a recipe to cause that
single recipe to honor the lock request and then carry-on as normal with that
whole recipe, sort of similar to how a $(MAKE) appearing anywhere within the
recipe gets special treatment.  And I do mean a single recipe, not all the
recipes in a given rule.

So that's my design plan, feel free to shoot at it.  That which doesn't kill
it will only make it stronger.

And one more thing...just to keep to things sane and avoid the inevitable
dead-lock scenarios, an $(flock lock-name) would only belong to a particular
instance of $(MAKE), and could not be inherited by sub-make's nor shared with
other sibling make' at the same level.  It would be private and local to
whatever makefile it's declared in, and travels no further.  This might be a
bit limiting, but the cost of implementing it should be bearable this way.
I'm trying to keep this small, sane, and yet usable in the most common
situations.  Its not intended to provide a transactional database locking
mechanism...

Any thoughts, suggestions, or pointers (especially to where in the make source
tree I should begin with...) are appreciated.

Thanks,
Zoltan


-- 
     ###  Any similarity between my views and the truth is completely ###
     ###     coincidental, except that they are endorsed by NO ONE    ###