Re: Looking for help improving performance (advice or potential contract position)

[Phil Grabsky]

Thanks for the responses, Dan and Rakesh.  You're both absolutely right
that a test case is required to demonstrate the problem; I'll need to see
if I can put a working example together at a reasonable scale, and I think
it's likely that scale itself is part of the problem.  Our system uses
includes to incorporate around 1500 makefiles (mostly generated from a
template) and run them in a single non-recursive make that compiles around
31,000 objects for multiple targets.  There are few if any direct
dependencies between

In general terms, what I observe is that when we have individual tasks
taking more than a second or two Make has no problem keeping up and will
maintain around 1000-2000 active jobs, but when jobs are under a second
it's only able to keep 10-20 jobs running in parallel, resulting in idle
workers.  I'll start with a simple test case of defining a makefile with
30,000 null jobs and see how long it takes make to get them all queued up,
then start layering in more complexity and see what I can come up with.

Thanks also to those who contacted me directly with some alternate sites to
post bounties or look for candidates; Here's the list of suggestions in
case anyone stumbles across this thread with a similar requirement later:
https://freedomsponsors.org/
https://www.bountysource.com/
https://www.fsf.org/resources/jobs


On Thu, Oct 2, 2014 at 4:21 AM, Rakesh Sharma <address@hidden> wrote:

> From your description what seems to be ailing your build system
> is the architecture of your makefile itself.
>
> Without any clues as to what ails your build (since you didnt provide the
> makefile), you need to make your makefile non-recursive in order to reap
> the speed dividends. The changes that you talk about (simply-expanded
> variables, no shell commands, etc.) go only so far in gaining speed.
>
> For more help you can look up the Miller paper on non-recursive makefiles
> which
> explains the reasons for slow/inaccurate builds & also how to overcome
> them.
>
> HTH
>
> -Rakesh
>
> > Date: Wed, 1 Oct 2014 13:44:02 -0400
> > Subject: Looking for help improving performance (advice or potential
> contract position)
> > From: address@hidden
> > To: address@hidden
>
> >
> > Greetings all,
> > We've been making an effort to optimize our builds and we seem to be
> > hitting the limit of what we can accomplish with makefile cleanup. I've
> > turned around some significant improvements by using techniques like
> > simplifying makefiles where possible, converting recursively-expanded
> > variables to simple variables, eliminating shell commands, etc, but we're
> > still seeing slow performance from Make itself.
> >
> > Our build is extremely parallelizable and we run with -j (unlimited) to
> run
> > as many things as possible in parallel. We use a home-grown job
> > distribution tool to send individual tasks to clients on other machines
> for
> > execution, so there's very little load on the machine running Make
> itself.
> > What we see is that if we have more than 30 or so clients connected to
> the
> > build Make is often unable to generate tasks fast enough to keep them
> busy.
> >
> > We brought in Electric Cloud to do an evaluation of their make-compatible
> > accelerator tool and found that it was able to instantly saturate as many
> > workers as we could throw at it and keep them busy for the duration of
> the
> > build, cutting the overall time nearly in half. Unfortunately, their
> > approach introduces a number of changes that would require significant
> > reworking of our build system, and more importantly they want quite a lot
> > of money for it (enough to hire additional senior developers for our
> team).
> >
> > Our management doesn't want us taking time to learn the internals of make
> > and see if we can improve it ourselves, but they are willing to throw
> money
> > at the problem, either by offering some sort of donation/bounty for
> > performance improvements or directly hiring an expert on a contract
> basis;
> > any improvements achieved would of course be submitted back to the
> > community as patches.
> >
> > Is there an established place to post this sort of offer?
> fossfactory.org
> > seems like what I'm looking for, but it doesn't look like it's terribly
> > active.
> > _______________________________________________
> > Help-make mailing list
> > address@hidden
> > https://lists.gnu.org/mailman/listinfo/help-make
>