RE: Question about error detection

[Cooper, Robert (Ftg Ops)]

I must express my profound thanks to you, first and foremost.  Not only have 
you provided the solution for the error detection issue, but your comment on 
parallelism led to some amazing results and also cleared up other related 
puzzles concerning build techniques which seemed to work in one context and 
fail in another.  I have always been interest in performance and throughput, 
but having inherited a fairly large build environment about a year ago my task 
was to maintain stability, support the environment, and begin the migration to 
Git.

Below are follow-up questions, but first a quick summary of the performance 
improvements.

To give you a rough idea of the performance improvements achieved with a 
variety of changes (some having already been made), I can attest to the 
following for one of the more complex applications:

•       After having already shaved significant time from the total, we were 
running the shared library build, editing the application configuration, then 
building the application 4 times (once per target environment) and it took 33 
minutes plus the time to perform the edit in between.
•       In the new build environment I have made it unnecessary to build the 
applications 4 times since the build contains all environmental configuration 
elements in one place.  If that were true in the old environment it would still 
take approximately 18 minutes of the 33 minutes above.
•       In the new environment I have made the shared library build a substep 
of each application build, plus for each one I am only building the specific 
subdirectories required and only building 32 bit or 64 bit where required.  The 
shared library has 30 subdirectories, and we used to build all of them for both 
32 bit and 64 bit.  Now in some cases I only build 8 of the subdirectories, and 
the most I have encountered so far is 16.  Adding in the parallel processing, 
the above build is taking about 2.5 minutes. Without the parallel processing I 
believe it was running for approximately 5-6 minutes.

I have two areas of discussion and questions I’d like to pursue here.

1)  Style of these builds – makes the real work appear to be a dependency 
instead of the work itself

I end up with many build commands that look somewhat like the ones below.  As 
you can see at the end, the two recipes basically do all their work as if it 
was a prerequisite dependency rather than the actual build itself.  But it 
works quite nicely, do you see any problems with this approach?

# the complete list of directories

BLD_SRC_DIRS  := dir1 dir2 dir3 dir4 dir5 dir6 dir7

# the real build command

$(BLD_SRC_DIRS) :
        $(QUIET) cd src ; $(MAKE) -j --directory=$(@) -f Makefile.gnu 
$(SHLIB-BLD) VERSION=$(VERSION) VERBOSE=$(VERBOSE)

# per-project lists of directories to build

BLD_DIRS_PROJECT1  := dir1 dir2 dir3

BLD_DIRS_PROJECT2  := dir1 dir2 dir4 dir5

# per-project recipes

build-project1 : $(BLD_DIRS_PROJECT1)
        $(QUIET) echo "SPM:  Finished building PROJECT1 64-bit only shlib 
components."

build-project2 : $(BLD_DIRS_PROJECT2)
        $(QUIET) echo "SPM:  Finished building PROJECT2 64-bit only shlib 
components."

2) Special case of above – requires a mix of 32 bit and 64 bit builds.

One project has several build targets, some of which require 32 bit builds 
because a vendor package does not support 64 bit.  Directory by directory, the 
build requirements turned out to be 32-bit only, 64-bit only, or both.  I 
created recipes called build (which builds both), build32 and build64 to 
support this.  I’m just looking for any comments or suggestion you may have 
concerning this approach – it does work, and it is quite clear what it is 
doing.  I imagine I can make the 3 steps in the new recipe below also run in 
parallel with one more level of refactoring if I can map the input arguments in 
the make syntax, but the benefit is likely to be incremental.  Each of the 3 
steps does use parallel processing, but with respect to each other they run 
sequentially as configured below.

build-app1-both : $(BLD_DIRS_APP1_BOTH)
        $(QUIET) echo "SPM:  Finished building APP1 32/64-bit shlib components."

build-app1-64 : $(BLD_DIRS_APP1_64)
        $(QUIET) echo "SPM:  Finished building APP1 64-bit only shlib 
components."

build-app1-32 : $(BLD_DIRS_APP1_32)
        $(QUIET) echo "SPM:  Finished building APP1 32-bit only shlib 
components."

build-app1 :
        $(QUIET) echo "SPM:  Starting build of APP1 shlib 32 bit components."
        $(MAKE) -f Makefile.gnu SHLIB-BLD=build32 build-app1-32 
VERSION=$(VERSION) VERBOSE=$(VERBOSE)
        $(QUIET) echo "SPM:  Starting build of APP1 shlib 64 bit components."
        $(MAKE) -f Makefile.gnu SHLIB-BLD=build64 build-app1-64 
VERSION=$(VERSION) VERBOSE=$(VERBOSE)
        $(QUIET) echo "SPM:  Starting build of APP1 shlib 32/64 bit components."
        $(MAKE) -f Makefile.gnu SHLIB-BLD=build build-app1-both 
VERSION=$(VERSION) VERBOSE=$(VERBOSE)

Thank you,
Bob



-----Original Message-----
From: Paul Smith [mailto:address@hidden
Sent: Thursday, October 02, 2014 3:35 PM
To: Cooper, Robert (Ftg Ops)
Cc: address@hidden
Subject: Re: Question about error detection

On Thu, 2014-10-02 at 15:11 -0400, Cooper, Robert (Ftg Ops) wrote:
> (I don’t see how one replies directly to the thread, so I clicked on
> the button to email you, apologies if that is inconvenient.)

You just reply to the mailing list (put it into the To: or CC: line of
your message).

> We are running Make 4.0 on Red Hat Linux --
>
> My primary makefile build recipe –
>
> build : $(VERSION_FILE) rel_dirs shlib vendor_libs
>         $(QUIET) echo "SPM:  Constructing release directory tree"
>         $(QUIET) for dir in $(BLD_DIRS); do \
>         $(MAKE) -f Makefile.gnu --directory $$dir build TYPE=$(TYPE) 
> VERSION=$(VERSION) VERBOSE=$(VERBOSE) ; \
>                  done
>         $(QUIET) $(FLAG_FULL_BUILD)

Yep, that's it.

You have a loop across all the subdirectories and in each one you run
make.  But you never check whether each make succeeded or failed, so any
make failure in a directory other than the last one will be ignored.
Only failures in the very last directory will be seen.

Make just hands the entire for-loop to the shell and waits to see what
the shell exits with; it's up to you to make sure it exits with an error
code if there's an error.

Use something like this instead:

  for dir in $(BLD_DIRS); do \
      $(MAKE) -f ... || exit 1; \
  done

Note the addition of the "|| exit 1" so that if that make command fails
we exit immediately with an error.

In general, doing loops in a shell like this is very un-make-like.  For
example, if you wanted to use parallel build support it won't work as
well as it could here: each of those sub-makes will be run serially by
the shell regardless of how much parallelism you enable (inside the
sub-make you'll get parallelism, of course).  Also, if you run make with
"-k" (keep going) it won't do what you want here, because the loop exits
out always the first time it hits an error.

This can be fixed, but I won't go into it unless you're interested.