[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [coreutils] cp --parents parallel
From: |
Bob Proulx |
Subject: |
Re: [coreutils] cp --parents parallel |
Date: |
Tue, 19 Oct 2010 10:53:49 -0600 |
User-agent: |
Mutt/1.5.20 (2009-06-14) |
Rob Gom wrote:
> Bob Proulx wrote:
> > And the last one copied wins? Okay.
>
> Well, that's not the case. There's make called. Make spawns some
> submakes. Every submake calls target which copies files (--parents
> --update).
> Every submake has its own files list, so only one cp is handling any
> given file. But they share some parts of directory structure. Exactly
> as in my example.
The --parents problem can be fixed by using 'mkdir -p' before cp.
> --update is needed because of rerunning make - in such case I would
> need cp to copy only newer files (optimisation).
I don't follow that yet. Make itself should be looking at timestamps
and updating targets that need updating. It shouldn't be updating
targets that don't need updating. If --update is in use then that
implies that targets are getting made multiple times?
> > You could easily call 'mkdir -p $(TARGET)' before calling cp, remove
> > the --parents, and avoid the problem.
>
> I can't use mkdir -p $(TARGET), because FILES_LIST:
> - is a list of files
> - with paths
> Example:
> TARGET=/tmp/a
> FILES_LIST=b/c/d/e.txt b/d/f.txt b/g.txt c/h.txt
> In such case I would need a rule to create not only /tmp/a, but
> /tmp/a/b/c/d, /tmp/a/b/d, /tmp/a/c. Then it gets a bit more
> complicated.
Something like this then?
TARGET=/tmp/a
FILES_LIST=b/c/d/e.txt b/d/f.txt b/g.txt c/h.txt
target:
for p in $(FILES_LIST) ; do p=$$(dirname $$p); mkdir -v -p
$(TARGET)/$$p; done
cp -v --update --parents $(FILES_LIST) $(TARGET)
That will output:
mkdir: created directory `/tmp/a'
mkdir: created directory `/tmp/a/b'
mkdir: created directory `/tmp/a/b/c'
mkdir: created directory `/tmp/a/b/c/d'
mkdir: created directory `/tmp/a/b/d'
mkdir: created directory `/tmp/a/c'
cp -v --update --parents b/c/d/e.txt b/d/f.txt b/g.txt c/h.txt /tmp/a
`b/c/d/e.txt' -> `/tmp/a/b/c/d/e.txt'
`b/d/f.txt' -> `/tmp/a/b/d/f.txt'
`b/g.txt' -> `/tmp/a/b/g.txt'
`c/h.txt' -> `/tmp/a/c/h.txt'
> Yes, I could handle that, but then I would have to rely on mkdir -p
> instead of relying of cp --parents (multiple processes could invoke
> mkdir -p with common directory root).
Correct that you would be relying upon 'mkdir -p'. But 'mkdir -p' has
been around for a long time, is quite portable, and has been
standardized by POSIX. True that it wasn't in V7 and isn't ultimately
portable. But it is undoubtedly more portable than 'cp --parents'.
There is an autoconf macro to deal with older systems if needed.
> I wrote first email with hope that I missed something obvious, which
> doesn't seem to be the case. I will rewrite the rules to use mkdir
> instead. Thank you for your opinions.
Good luck!
Bob