Re: Gnu make option to fix parallel multiple ar on a shared archive

[Philip Guenther]

On Sat, Aug 10, 2024 at 8:16 PM Navin P <navinp0304@gmail.com> wrote:
>
> On Sun, Aug 11, 2024, 06:29 Kaz Kylheku <kaz@kylheku.com> wrote:
>
> > Your makefile is broken, or at least not suitable for parallel builds.
> >
> > You should add .NOTPARALLEL: to it. That's the minimal change that will
> > fix it. A makefile with the .NOTPARALLEL: target will not be parallelized
> > even if -j is requested on the command line or via the environment.
> >
> > Why it's broken is that you have multiple rules that update the same
> > object.
> >
> > An archive should be built by exactly one rule which has all the
> > prerequisites that are going into the archive.
> >
> > On August 10, 2024 4:04:13 AM PDT, Navin P <navinp0304@gmail.com> wrote:
> > >Hi,
> > >   I ran multiple instances of ar during a parallel build and they were
> > >updating different object files using ar rv sometimes the same object file
> > >again.
> > >
> > >When i run make -j1 i find all the object files  libx.a but when i do make
> > >-j8 i find some missing object files inside libx.a.
> > >
> > >Then i looked at ar.c and found that there is no synchronization between 2
> > >instances of ar on same library. The problem is that makefile is generated
> > >and  there are lots of bash scripts doing that.
> > >
> > >Is there any way to fix the problem?
> > >1. Without changing makefile
> > >2. With changing the makefile in long term.
> > >
> >
>
> I was looking at
> https://stackoverflow.com/questions/8824969/how-do-i-avoid-concurrency-problems-when-using-gnu-make-for-parallel-builds-of-a
>
> AR := flock make.lock $(AR)
>
>
> What are the problems with using flock ?

I think the correctness concern is that this provides mutual exclusion
for ar only.  Other programs may be reading the .a, including make
itself.  If your system's ar does atomic updates (write new temp file,
rename over existing .a) then the readers should have a
self-consistent view and this should work Just Fine.  If not, then
depending on exactly how it does update then make may get confused
about the .a contents.


The performance concern is two-fold:

1) ar's normal operation includes rebuilding the symbol table after
each change.  Adding objects to a .a with separate ar invocations
results in O(N^2) symbol operations.

2) make's parallel operation isn't aware of the "these operations will
be serialized" behavior, so make may schedule multiple 'invoke rule
using $(AR)' recipes concurrently despite them being serialized,
resulting in reduced parallelism vs the "build all of the out-of-date
objects then insert them all at once" rule setup.


Philip Guenther