Re: [coreutils] cp --parents parallel

[Bob Proulx]

Rob Gom wrote:
> Bob Proulx wrote:
> > And the last one copied wins?  Okay.
> 
> Well, that's not the case. There's make called. Make spawns some
> submakes. Every submake calls target which copies files (--parents
> --update).
> Every submake has its own files list, so only one cp is handling any
> given file. But they share some parts of directory structure. Exactly
> as in my example.

The --parents problem can be fixed by using 'mkdir -p' before cp.

> --update is needed because of rerunning make - in such case I would
> need cp to copy only newer files (optimisation).

I don't follow that yet.  Make itself should be looking at timestamps
and updating targets that need updating.  It shouldn't be updating
targets that don't need updating.  If --update is in use then that
implies that targets are getting made multiple times?

> > You could easily call 'mkdir -p $(TARGET)' before calling cp, remove
> > the --parents, and avoid the problem.
> 
> I can't use mkdir -p $(TARGET), because FILES_LIST:
> - is a list of files
> - with paths
> Example:
> TARGET=/tmp/a
> FILES_LIST=b/c/d/e.txt b/d/f.txt b/g.txt c/h.txt
> In such case I would need a rule to create not only /tmp/a, but
> /tmp/a/b/c/d, /tmp/a/b/d, /tmp/a/c. Then it gets a bit more
> complicated.

Something like this then?

TARGET=/tmp/a
FILES_LIST=b/c/d/e.txt b/d/f.txt b/g.txt c/h.txt
target:
        for p in $(FILES_LIST) ; do p=$$(dirname $$p); mkdir -v -p 
$(TARGET)/$$p; done
        cp -v --update --parents $(FILES_LIST) $(TARGET)

That will output:

  mkdir: created directory `/tmp/a'
  mkdir: created directory `/tmp/a/b'
  mkdir: created directory `/tmp/a/b/c'
  mkdir: created directory `/tmp/a/b/c/d'
  mkdir: created directory `/tmp/a/b/d'
  mkdir: created directory `/tmp/a/c'
  cp -v --update --parents b/c/d/e.txt b/d/f.txt b/g.txt c/h.txt /tmp/a
  `b/c/d/e.txt' -> `/tmp/a/b/c/d/e.txt'
  `b/d/f.txt' -> `/tmp/a/b/d/f.txt'
  `b/g.txt' -> `/tmp/a/b/g.txt'
  `c/h.txt' -> `/tmp/a/c/h.txt'

> Yes, I could handle that, but then I would have to rely on mkdir -p
> instead of relying of cp --parents (multiple processes could invoke
> mkdir -p with common directory root).

Correct that you would be relying upon 'mkdir -p'.  But 'mkdir -p' has
been around for a long time, is quite portable, and has been
standardized by POSIX.  True that it wasn't in V7 and isn't ultimately
portable.  But it is undoubtedly more portable than 'cp --parents'.
There is an autoconf macro to deal with older systems if needed.

> I wrote first email with hope that I missed something obvious, which
> doesn't seem to be the case. I will rewrite the rules to use mkdir
> instead. Thank you for your opinions.

Good luck!

Bob