Re: [coreutils] cp --parents parallel

[Bob Proulx]

Rob Gom wrote:
> Regarding my example - I am aware of bashism. That was only an example
> and it was more convenient for me to use bash features.

But you didn't use any bash features that I noticed.

> As for the real case - I use it inside makefiles. I copy many
> directory structures into single root.

And the last one copied wins?  Okay.

I am inclined to suggest using 'rsync' if --update is important.  The
way that rsync updates the target is different from the way that cp
updates the target.  The cp command writes it in place.  The rsync
command uses a temporary file and a rename to ensure that the target
is never available in a half written state.  Also I am not convinced
that cp is completely race-condition safe when multiple cp processes
are writing to the file at the same time.  Could it get duplicated
data in the file?  I would need to look.

> If --parents itself works as expected, that will be the easiest
> solution.

I think it would be reasonable to make --parents work the same as it
does for 'mkdir --parents' and not complain if the internal mkdir
fails because the directory already exists.  But even if that is
patched it will be years before the release containing it flows down
to most installations where you can count on having it.  Your 7.4 copy
was released on 2009-05-07 and yet you still have it in place and
probably will for a while I would imagine.  Current is version 8.6
released 2010-10-15.  Therefore it would still be wise to avoid the
race and use techniques that don't exhibit the problem.  The example
code I posted showed one such way.

> Generally it looks like:
> target:
>     cp --parents --update $(FILES_LIST) $(TARGET)

You could easily call 'mkdir -p $(TARGET)' before calling cp, remove
the --parents, and avoid the problem.

> (pseudo code, not working). As you can see, it's more complicated than
> simple cp, so I wanted to avoid that.

You could very easily convert to making the directory first and then
copying the files into place however.  That wouldn't change what you
were doing very much at all.

If all copies are equivalent (since you have a race and any of the
subprocesses might update it) then it would seem to be better to do it
once only instead of many times in parallel and discarding the first
ones to be copied and keeping the last.  You would probably speed up
your operation too since it wouldn't be doing unnecessary work.

> By the way, I have just reproduced the issue on Debian with coreutils
> 8.5. To my surprise, it's much less frequent, though. However this is
> a completely different machine - that may explain the behaviour (I
> haven't seen relevant change in general release notes between 7.4 and
> 8.5).

Since it depeneds upon the order of execution the differences in
machines could cause quite a bit of change is the way the race
conditions are won.  You could see entirely different behavior.

Bob