Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?

[James Youngman]

On Wed, Nov 12, 2008 at 4:01 PM, Phillip Susi <address@hidden> wrote:
> James Youngman wrote:
>>
>> This version should be race-free:
>>
>> find -type f -print0 |
>>     xargs -0 -n 8 --max-procs=16 md5sum >> ~/md5sums 2>&1
>>
>> I think that writing into a pipe should be OK, since pipes are
>> non-seekable.  However, with pipes in this situation you still have a
>> problem if processes try to write more than PIPE_BUF bytes.
>
> You aren't using a pipe there.

I know this.   My point was that >> ensures that the file is opened in
O_APPEND mode, but that the equivalent xargs command with a pipe
instead of a file redirection is safe.

> What you are doing is having the shell open
> the file, then the md5sum processes all inherit that fd so they all share
> the same offset.  As long as they write() the entire line at once, the file
> pointer will be updated atomically for all processes and the lines from each
> process won't clobber each other.
>
>