[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
How to restrict parallelism in make? [PATCH]
|
From: |
Alexey Neyman |
|
Subject: |
How to restrict parallelism in make? [PATCH] |
|
Date: |
Fri, 29 Apr 2005 13:28:17 +0400 |
|
User-agent: |
KMail/1.6.2 |
Hi,
I have come across the following problem with parallel jobs in make. I
have a project that uses recursive build. I'd like to have
subdirectories being built before the local targets. In a sequential
Makefile, one would do the following:
<<<<
.PHONY: all all-subdir all-local
all: all-subdir all-local
all-subdir:
# visit the subdirectories and run "make all" there
all-local: $(whatever local targets are)
>>>>
In parallel make, however, the 'all-local' and 'all-subdir' targets
will be made in parallel. Note that making 'all-local' depend on
'all-subdir' won't help: the prerequisites of 'all-local' will still
be made in parallel with 'all-subdir'.
Well, this could be worked around by making 'all-subdir' an order-only
prerequisite to each and every target that 'all-local' depends upon,
and their prerequisites, and so on... Though I believe this is rather
counternatural, it helps.
And the real problems arise when I try to add the 'install' target.
Similarly, I'd like the 'install' target to walk into the
subdirectories first and then perform local activities:
<<<<
install: install-subdir install-local
>>>>
But if I add the install-subdir as an order-only prerequisite to all
targets, as described above, this will cause "make install" executed
in the subdirectories even in case of "make all" in the top-level
directory. Well, this could be worked around by having
'install-subdir' target depend on 'all', but this way the
subdirectories will be traversed twice: once for "make all" and once
again for "make install". Alternatively, one could serialize the
targets artificially (at a cost of extra instance of make running):
<<<<
all:
$(MAKE) all-subdir
$(MAKE) all-local
install:
$(MAKE) install-subdir
$(MAKE) install-local
>>>>
An elegant solution to this problem has long been available in BSD
version of make (pmake):
http://www.freebsd.org/cgi/man.cgi?query=make&apropos=0&sektion=0&manpath=FreeBSD+5.3-stable&format=html
There, the special '.WAIT' target when listed as a dependency, acts as
a "serializer": all dependencies _before_ it in the list are finished
before any of those _after_ .WAIT are even started to be updated.
With this, the above requirements could be supported easily:
<<<<
all: all-subdir .WAIT all-local
install: install-subdir .WAIT install-local
>>>>
I therefore decided to implement it in GNU make. The suggested patch
is attached. I also added the ability to specify the whole target as
requiring sequential update by specifying that target as a dependency
for the .NOTPARALLEL target (without prerequisites, it disables
parallelism at all).
Also attached is a test Makefile and the results of the tests. Run
with "make -j WAIT=.WAIT <target>", where target is one of the "all",
"all2", "all3" (they test the .WAIT behavior for implicit
dependencies, explicit ones and .NOTPARALLEL, respectively).
There is an alternative in the SCO version of make available (.MUTEX):
http://uw713doc.sco.com/en/man/html.1/make.1.html
However, the BSD variant is more flexible, so I chose that.
Regards,
Alexey.
--
For mundane material concerns, seek the help of the one-eyed beings.
-- Pkunks, SC2
all: a b $(WAIT) c d s-e s-f
all2: a b c $(WAIT) d e $(WAIT) f g h $(WAIT) i $(WAIT) $(WAIT) j k
all3: a b sub3 i j
.PHONY: all1 all2 all3 sub3 clean
sub3: c d e f g h
.NOTPARALLEL: sub3
clean:
rm -f [a-k] [stu]-*
a b c d e f g h i j k:
@echo Starting update of address@hidden; sleep 1; touch $@; echo
Finished update of address@hidden
s-%: u-% $(WAIT) t-%
@echo Starting update of address@hidden; sleep 1; touch $@; echo
Finished update of address@hidden
t-%:
@echo Starting update of address@hidden; sleep 1; touch $@; echo
Finished update of address@hidden
u-%:
@echo Starting update of address@hidden; sleep 1; touch $@; echo
Finished update of address@hidden
$ ../make/make clean
rm -f [a-k] [stu]-*
$ ../make/make -j all WAIT=.WAIT
Starting update of a...
Starting update of b...
Finished update of a.
Finished update of b.
Starting update of c...
Starting update of d...
Starting update of u-f...
Starting update of u-e...
Finished update of d.
Finished update of u-f.
Finished update of c.
Finished update of u-e.
Starting update of t-f...
Starting update of t-e...
Finished update of t-f.
Finished update of t-e.
Starting update of s-f...
Starting update of s-e...
Finished update of s-f.
Finished update of s-e.
rm t-f u-f t-e u-e
$ ../make/make clean
rm -f [a-k] [stu]-*
$ ../make/make -j all2 WAIT=.WAIT
Starting update of a...
Starting update of b...
Starting update of c...
Finished update of a.
Finished update of b.
Finished update of c.
Starting update of d...
Starting update of e...
Finished update of d.
Finished update of e.
Starting update of f...
Starting update of g...
Starting update of h...
Finished update of f.
Finished update of g.
Finished update of h.
Starting update of i...
Finished update of i.
Starting update of j...
Starting update of k...
Finished update of k.
Finished update of j.
$ ../make/make clean
rm -f [a-k] [stu]-*
$ ../make/make -j all3 WAIT=.WAIT
Starting update of a...
Starting update of b...
Starting update of c...
Starting update of i...
Starting update of j...
Finished update of a.
Finished update of b.
Finished update of c.
Finished update of i.
Starting update of d...
Finished update of j.
Finished update of d.
Starting update of e...
Finished update of e.
Starting update of f...
Finished update of f.
Starting update of g...
Finished update of g.
Starting update of h...
Finished update of h.
Index: default.c
===================================================================
RCS file: /cvsroot/make/make/default.c,v
retrieving revision 1.44
diff -u -u -r1.44 default.c
--- default.c 16 May 2004 19:16:53 -0000 1.44
+++ default.c 29 Apr 2005 08:38:51 -0000
@@ -532,7 +532,7 @@
char *p = default_suffixes;
suffix_file->deps = (struct dep *)
multi_glob (parse_file_seq (&p, '\0', sizeof (struct dep), 1),
- sizeof (struct dep));
+ sizeof (struct dep), NULL);
(void) define_variable ("SUFFIXES", 8, default_suffixes, o_default, 0);
}
}
Index: dep.h
===================================================================
RCS file: /cvsroot/make/make/dep.h,v
retrieving revision 1.19
diff -u -u -r1.19 dep.h
--- dep.h 13 Apr 2005 03:16:33 -0000 1.19
+++ dep.h 29 Apr 2005 08:38:51 -0000
@@ -41,6 +41,7 @@
unsigned int changed : 8;
unsigned int ignore_mtime : 1;
unsigned int need_2nd_expansion : 1;
+ unsigned int wait : 1;
};
@@ -53,7 +54,7 @@
};
-extern struct nameseq *multi_glob PARAMS ((struct nameseq *chain, unsigned int
size));
+extern struct nameseq *multi_glob PARAMS ((struct nameseq *chain, unsigned int
size, int *seen_special));
#ifdef VMS
extern struct nameseq *parse_file_seq ();
#else
Index: file.c
===================================================================
RCS file: /cvsroot/make/make/file.c,v
retrieving revision 1.78
diff -u -u -r1.78 file.c
--- file.c 13 Apr 2005 03:16:33 -0000 1.78
+++ file.c 29 Apr 2005 08:39:23 -0000
@@ -414,6 +414,28 @@
f->intermediate = 1;
}
+/* Notice special dependencies. */
+void
+notice_special_deps(struct dep **d_ptr)
+{
+ struct dep *d;
+
+restart_loop:
+ for (; *d_ptr; d_ptr = &(*d_ptr)->next)
+ {
+ d = *d_ptr;
+ if (!strcmp(d->name, ".WAIT"))
+ {
+ free(d->name);
+ if (d->next)
+ d->next->wait = 1;
+ *d_ptr = d->next;
+ free(d);
+ goto restart_loop;
+ }
+ }
+}
+
/* Expand and parse each dependency line. */
static void
expand_deps (struct file *f)
@@ -422,6 +444,7 @@
struct dep *new = 0;
struct dep *old = f->deps;
unsigned int last_dep_has_cmds = f->updating;
+ int seen_special = 0;
int initialized = 0;
f->updating = 0;
@@ -457,7 +480,7 @@
new = (struct dep *)
multi_glob (
parse_file_seq (&p, '|', sizeof (struct dep), 1),
- sizeof (struct dep));
+ sizeof (struct dep), &seen_special);
if (*p)
{
@@ -473,12 +496,18 @@
*d_ptr = (struct dep *)
multi_glob (
parse_file_seq (&p, '\0', sizeof (struct dep), 1),
- sizeof (struct dep));
+ sizeof (struct dep), &seen_special);
for (d1 = *d_ptr; d1 != 0; d1 = d1->next)
d1->ignore_mtime = 1;
}
+ /* Before dependencies are entered as files, treat special
+ dependencies. Need to do this before entering them as files
+ as treatment might involve deletion of a dependency. */
+ if (seen_special)
+ notice_special_deps(&new);
+
/* Enter them as files. */
for (d1 = new; d1 != 0; d1 = d1->next)
{
@@ -538,7 +567,7 @@
{
struct file *f;
struct file *f2;
- struct dep *d;
+ struct dep *d, *d2;
struct file **file_slot_0;
struct file **file_slot;
struct file **file_end;
@@ -646,9 +675,20 @@
if (f != 0 && f->is_target)
posix_pedantic = 1;
+ /* .NOTPARALLEL with deps makes these targets non-parallel (i.e. all
+ their dependencies are serialized). Without deps, it forbids
+ parallel execution at all. */
f = lookup_file (".NOTPARALLEL");
if (f != 0 && f->is_target)
- not_parallel = 1;
+ {
+ if (f->deps == 0)
+ not_parallel = 1;
+ else
+ for (d = f->deps; d != 0; d = d->next)
+ for (f2 = d->file; f2 != 0; f2 = f2->prev)
+ for (d2 = f2->deps; d2 != 0; d2 = d2->next)
+ d2->wait = 1;
+ }
/* Remember that we've done this. */
snapped_deps = 1;
Index: filedef.h
===================================================================
RCS file: /cvsroot/make/make/filedef.h,v
retrieving revision 2.22
diff -u -u -r2.22 filedef.h
--- filedef.h 28 Feb 2005 07:48:22 -0000 2.22
+++ filedef.h 29 Apr 2005 08:39:23 -0000
@@ -113,6 +113,7 @@
extern void notice_finished_file PARAMS ((struct file *file));
extern void init_hash_files PARAMS ((void));
extern char *build_target_list PARAMS ((char *old_list));
+extern void notice_special_deps PARAMS ((struct dep **d));
#if FILE_TIMESTAMP_HI_RES
# define FILE_TIMESTAMP_STAT_MODTIME(fname, st) \
Index: function.c
===================================================================
RCS file: /cvsroot/make/make/function.c,v
retrieving revision 1.87
diff -u -u -r1.87 function.c
--- function.c 4 Mar 2005 12:52:32 -0000 1.87
+++ function.c 29 Apr 2005 08:39:24 -0000
@@ -353,7 +353,7 @@
That would break examples like:
$(patsubst ./%.c,obj/%.o,$(wildcard ./?*.c)). */
0),
- sizeof (struct nameseq));
+ sizeof (struct nameseq), NULL);
if (result == 0)
{
Index: implicit.c
===================================================================
RCS file: /cvsroot/make/make/implicit.c,v
retrieving revision 1.47
diff -u -u -r1.47 implicit.c
--- implicit.c 13 Apr 2005 03:16:33 -0000 1.47
+++ implicit.c 29 Apr 2005 08:39:24 -0000
@@ -75,6 +75,7 @@
char *intermediate_pattern; /* pattern for intermediate file */
unsigned char had_stem; /* had % substituted with stem */
unsigned char ignore_mtime; /* ignore_mtime flag */
+ unsigned char wait;
};
static void
@@ -185,6 +186,28 @@
return beg;
}
+/* Handle special dependencies in implicit rules. */
+static void
+notice_special_ideps(struct idep **id_ptr)
+{
+ struct idep *id;
+
+restart_loop:
+ for (; *id_ptr; id_ptr = &(*id_ptr)->next)
+ {
+ id = *id_ptr;
+ if (!strcmp(id->name, ".WAIT"))
+ {
+ free(id->name);
+ if (id->next)
+ id->next->wait = 1;
+ *id_ptr = id->next;
+ free(id);
+ goto restart_loop;
+ }
+ }
+}
+
/* Search the pattern rules for a rule with an existing dependency to make
FILE. If a rule is found, the appropriate commands and deps are put in FILE
and 1 is returned. If not, 0 is returned.
@@ -261,6 +284,10 @@
that is not just `%'. */
int specific_rule_matched = 0;
+ /* Nonzero if we saw possibly special dependencies (e.g. .WAIT) while
+ parsing the dependencies for that rule. */
+ int seen_special;
+
register unsigned int i = 0; /* uninit checks OK */
register struct rule *rule;
register struct dep *dep, *expl_d;
@@ -453,6 +480,9 @@
pattern_search won't try to use it. */
rule->in_use = 1;
+ /* This rule hasn't encountered special dependencies (so far). */
+ seen_special = 0;
+
/* From the lengths of the filename and the matching pattern parts,
find the stem: the part of the filename that matches the %. */
stem = filename
@@ -557,14 +587,14 @@
multi_glob (
parse_file_seq (&p2,
order_only ? '\0' : '|',
- sizeof (struct idep),
- 1), sizeof (struct idep));
+ sizeof (struct idep), 1),
+ sizeof (struct idep), &seen_special);
/* @@ It would be nice to teach parse_file_seq or
multi_glob to add prefix. This would save us
some reallocations. */
- if (order_only || add_dir || had_stem)
+ if (order_only || had_stem)
{
unsigned long l = lastslash - filename + 1;
@@ -609,6 +639,10 @@
file->stem = 0;
+ /* If there were special dependencies, handle them. */
+ if (seen_special)
+ notice_special_ideps(&deps);
+
/* @@ This loop can be combined with the previous one. I do
it separately for now for transparency.*/
@@ -836,6 +870,7 @@
dep = (struct dep *) xmalloc (sizeof (struct dep));
dep->ignore_mtime = d->ignore_mtime;
+ dep->wait = d->wait;
dep->need_2nd_expansion = 0;
s = d->name; /* Hijacking the name. */
d->name = 0;
@@ -908,7 +943,7 @@
{
struct dep *new = (struct dep *) xmalloc (sizeof (struct dep));
/* GKM FIMXE: handle '|' here too */
- new->ignore_mtime = 0;
+ new->ignore_mtime = new->wait = 0;
new->need_2nd_expansion = 0;
new->name = p = (char *) xmalloc (rule->lens[i] + fullstemlen + 1);
bcopy (rule->targets[i], p,
Index: main.c
===================================================================
RCS file: /cvsroot/make/make/main.c,v
retrieving revision 1.202
diff -u -u -r1.202 main.c
--- main.c 13 Apr 2005 03:16:33 -0000 1.202
+++ main.c 29 Apr 2005 08:39:24 -0000
@@ -2108,7 +2108,7 @@
ns = multi_glob (
parse_file_seq (&p, '\0', sizeof (struct nameseq), 1),
- sizeof (struct nameseq));
+ sizeof (struct nameseq), NULL);
/* .DEFAULT_TARGET should contain one target. */
if (ns->next != 0)
@@ -2124,7 +2124,7 @@
goals = (struct dep *) xmalloc (sizeof (struct dep));
goals->next = 0;
goals->name = 0;
- goals->ignore_mtime = 0;
+ goals->ignore_mtime = goals->wait = 0;
goals->need_2nd_expansion = 0;
goals->file = default_goal_file;
}
@@ -2290,7 +2290,7 @@
}
lastgoal->name = 0;
lastgoal->file = f;
- lastgoal->ignore_mtime = 0;
+ lastgoal->ignore_mtime = lastgoal->wait = 0;
lastgoal->need_2nd_expansion = 0;
{
Index: read.c
===================================================================
RCS file: /cvsroot/make/make/read.c,v
retrieving revision 1.143
diff -u -u -r1.143 read.c
--- read.c 13 Apr 2005 03:16:33 -0000 1.143
+++ read.c 29 Apr 2005 08:39:24 -0000
@@ -252,7 +252,7 @@
d->name = 0;
d->file = enter_file (*p);
d->file->dontcare = 1;
- d->ignore_mtime = 0;
+ d->ignore_mtime = d->wait = 0;
d->need_2nd_expansion = 0;
/* Tell update_goal_chain to bail out as soon as this file is
made, and main not to die if we can't make this file. */
@@ -366,6 +366,7 @@
read_makefiles = deps;
deps->name = 0;
deps->file = lookup_file (filename);
+ deps->ignore_mtime = deps->wait = 0;
if (deps->file == 0)
deps->file = enter_file (xstrdup (filename));
if (filename != ebuf.floc.filenm)
@@ -373,6 +374,7 @@
filename = deps->file->name;
deps->changed = flags;
deps->ignore_mtime = 0;
+ deps->wait = 0;
deps->need_2nd_expansion = 0;
if (flags & RM_DONTCARE)
deps->file->dontcare = 1;
@@ -807,7 +809,7 @@
files = multi_glob (parse_file_seq (&p2, '\0',
sizeof (struct nameseq),
1),
- sizeof (struct nameseq));
+ sizeof (struct nameseq), NULL);
free (p);
/* Save the state of conditionals and start
@@ -1002,7 +1004,7 @@
filenames = multi_glob (parse_file_seq (&p2, '\0',
sizeof (struct nameseq),
1),
- sizeof (struct nameseq));
+ sizeof (struct nameseq), NULL);
*p2 = ':';
if (!filenames)
@@ -2955,7 +2957,7 @@
that have room for additional info. */
struct nameseq *
-multi_glob (struct nameseq *chain, unsigned int size)
+multi_glob (struct nameseq *chain, unsigned int size, int *seen_special)
{
extern void dir_setup_glob ();
register struct nameseq *new = 0;
@@ -2983,6 +2985,10 @@
}
}
+ if (seen_special && old->name[0] == '.'
+ && isalpha (old->name[1]) && isupper (old->name[1]))
+ *seen_special = 1;
+
#ifndef NO_ARCHIVES
if (ar_name (old->name))
{
Index: remake.c
===================================================================
RCS file: /cvsroot/make/make/remake.c,v
retrieving revision 1.116
diff -u -u -r1.116 remake.c
--- remake.c 8 Apr 2005 12:51:20 -0000 1.116
+++ remake.c 29 Apr 2005 08:39:24 -0000
@@ -361,6 +361,15 @@
error (NILF, msg_parent, "*** ", file->name, file->parent->name, ".");
}
+/* Mark the dependencies as considered in this pass (to avoid skipping them
+ on the next pass). */
+static void
+consider_deps (struct dep *d)
+{
+ for (; d; d = d->next)
+ d->file->considered = considered;
+}
+
/* Consider a single `struct file' and update it as appropriate. */
static int
@@ -471,6 +480,13 @@
int maybe_make;
int dontcare = 0;
+ /* The code below will set the state to cs_deps_running. */
+ if (d->wait && running)
+ {
+ consider_deps (d);
+ break;
+ }
+
check_renamed (d->file);
mtime = file_mtime (d->file);
@@ -542,20 +558,28 @@
if (must_make || always_make_flag)
{
for (d = file->deps; d != 0; d = d->next)
- if (d->file->intermediate)
- {
- int dontcare = 0;
+ {
+ /* The code below will set the state to cs_deps_running. */
+ if (d->wait && running)
+ {
+ consider_deps(d);
+ break;
+ }
- FILE_TIMESTAMP mtime = file_mtime (d->file);
- check_renamed (d->file);
- d->file->parent = file;
+ if (d->file->intermediate)
+ {
+ int dontcare = 0;
- /* Inherit dontcare flag from our parent. */
- if (rebuilding_makefiles)
- {
- dontcare = d->file->dontcare;
- d->file->dontcare = file->dontcare;
- }
+ FILE_TIMESTAMP mtime = file_mtime (d->file);
+ check_renamed (d->file);
+ d->file->parent = file;
+
+ /* Inherit dontcare flag from our parent. */
+ if (rebuilding_makefiles)
+ {
+ dontcare = d->file->dontcare;
+ d->file->dontcare = file->dontcare;
+ }
dep_status |= update_file (d->file, depth);
@@ -585,7 +609,8 @@
if (!running)
d->changed = ((file->phony && file->cmds != 0)
|| file_mtime (d->file) != mtime);
- }
+ }
+ }
}
finish_updating (file);
Index: rule.c
===================================================================
RCS file: /cvsroot/make/make/rule.c,v
retrieving revision 1.37
diff -u -u -r1.37 rule.c
--- rule.c 13 Apr 2005 03:16:33 -0000 1.37
+++ rule.c 29 Apr 2005 08:39:24 -0000
@@ -205,7 +205,7 @@
deps = (struct dep *) xmalloc (sizeof (struct dep));
deps->next = 0;
deps->name = depname;
- deps->ignore_mtime = 0;
+ deps->ignore_mtime = deps->wait = 0;
deps->need_2nd_expansion = 0;
}
@@ -365,6 +365,7 @@
{
register struct rule *r;
char *ptr;
+ int seen_special = 0;
r = (struct rule *) xmalloc (sizeof (struct rule));
@@ -390,7 +391,10 @@
ptr = p->dep;
r->deps = (struct dep *) multi_glob (parse_file_seq (&ptr, '\0',
sizeof (struct dep), 1),
- sizeof (struct dep));
+ sizeof (struct dep), &seen_special);
+
+ if (seen_special)
+ notice_special_deps(&r->deps);
if (new_pattern_rule (r, 0))
{
- How to restrict parallelism in make? [PATCH],
Alexey Neyman <=