[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Thoughts on limiting the parallel build load, suggestion for a new "-j a
From: |
R. Diez |
Subject: |
Thoughts on limiting the parallel build load, suggestion for a new "-j auto" option |
Date: |
Tue, 21 Feb 2012 09:27:36 +0000 (GMT) |
Hi all:
I recently came across a build test script that launched "make -j" with no
limits, which consumed all available RAM and ground my Linux system to a halt.
I had to power the computer off, there was nothing else I could do.
After this experience, I believe that the default limitless behaviour of -j is
too dangerous. Even if your PC does not end up dying, GNU Make may
inadvertently consume too many resources. An explicit "-j infinite" would be a
better option, if anybody really needs something like that.
I recently came across option -l , which limits the amount of parallel tasks
based on the system's average load. However, this flag does not seem safe
enough, as, according to Wikipedia, not all Unix systems include in the average
load those processes currently waiting for disk I/O. Besides, the maximum
average load one would almost certainly want to use depends on the number of
CPUs, so the calling script (or user) has to find out how many CPUs there are.
How you find out may also depend on the operating system underneath, so
everybody gets a little extra work every time.
I am writing a build test framework myself, and I have been trying to
coordinate all sub-makes from a main makefile. The top-level script decides how
many parallel processes are allowed for the entire build and relies on
MAKEFLAGS in order to let all sub-makes talk to each other so as to limit the
overall load. Because different makefiles may require different GNU Make
options, I am filtering out all others and leaving just the parallel build
flags in place, like this:
export MAKEFLAGS="$(filter --jobserver-fds=%,$(MAKEFLAGS)) $(filter
-j,$(MAKEFLAGS))" && $(MAKE) ...etc...
By the way, option --jobserver-fds is not documented, but I think it should be.
As you can see, the user may need to filter it out manually after all.
The trouble with this MAKEFLAGS technique is that I often come across some
third-party script which insists on calculating and setting its own -j value,
rendering my coordination efforts useless. When this happens, I get warnings
like this:
warning: -jN forced in submake: disabling jobserver mode
Needless to say, most heuristics to calculate the -j value are as lame as mine
(see below). When writing build scripts, nobody seems to have much time left
for finding out how to retrieve the relevant system information
in bash/perl/whatever in a portable way and then calculate a good -j value out
of it.
I have been thinking about the best way to overcome such parallel woes, and I
wanted to share this thought with you all. How about adding to GNU Make a new
-j parameter like this:
make -j auto
The behaviour of "-j auto" could be as follows:
1) If MAKEFLAGS specifies -j and --jobserver-fds , then use those settings (no
warning required).
2) Otherwise, calculate the maximum number of parallel tasks with some trivial
heuristic based on the number of CPUs and/or the system load. I'm using <CPU
count> + 1 at the moment, but I'm sure there are better guesses.
I could think of several alternative heuristics:
make -j auto-system-load # Use -l <CPU count + 0.5>
make -j auto-processor-count # Use -j <CPU count + 1>
I guess most people would then end up using some "-j auto" variant, in order to
avoid overloading or underloading the system without having to implement their
own heuristics. That way, a top-level script will be much more likely to
succeed at setting a global limit when launching third-party sub-makes in
parallel.
Please copy me on the answers, as I'm not on this list.
Thanks,
R. Diez
- Thoughts on limiting the parallel build load, suggestion for a new "-j auto" option,
R. Diez <=