[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH] md5: accepts a new --threads option
From: |
Jim Meyering |
Subject: |
Re: [PATCH] md5: accepts a new --threads option |
Date: |
Wed, 21 Oct 2009 13:19:43 +0200 |
Pádraig Brady wrote:
> Jim Meyering wrote:
>> Pádraig Brady wrote:
>>> Pádraig Brady wrote:
>>>> You wouldn't want multiple threads/processes fighting over
>>>> the disk head so you would do something like:
>>>>
>>>> find /disk1 | xargs md5sum & find /disk2 | xargs md5sum
>>>>
>>>> Note if we're piping/redirecting the output of the above
>>>> then we must be careful to line buffer the output from md5sum
>>>> so that it's not interspersed. Hmm I wonder should
>>>> we linebuffer the output from *sum by default.
>>> In the attached patch, I've changed the default buffering
>>> to line buffered to address the above issue. For standard
>>> size files there is a 2% performance drop.
>>
>> Good catch.
>> It sounds like this fixes a real (albeit obscure) bug, so this
>> might deserve a NEWS item, though I admit it is borderline.
>
> Well it would easily be hit when one tries to parallelize the processes.
> So I'll add a NEWS item and a test along the lines of:
Thanks!
> (mkdir t && cd t && seq 100 | xargs touch)
> (find t t t t -type f | xargs -n100 -P4 md5sum) \
> | sed -n '/[0-9a-f]\{32\} /!p' | grep . >/dev/null && fail=1
Odd... that doesn't fail on any of the systems where I tried it:
rawhide, fedora 11, debian unstable.
- Re: [PATCH] core-count: A new program to count the number of cpu cores, (continued)
Re: [PATCH] md5: accepts a new --threads option, Giuseppe Scrivano, 2009/10/17
Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/20
- Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/20
- Re: [PATCH] md5: accepts a new --threads option, Jim Meyering, 2009/10/21
- Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/21
- Re: [PATCH] md5: accepts a new --threads option,
Jim Meyering <=
- Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/21
- Re: [PATCH] md5: accepts a new --threads option, Jim Meyering, 2009/10/21
- Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/21
RE: [PATCH] md5: accepts a new --threads option, Voelker, Bernhard, 2009/10/21
Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/22
Re: [PATCH] md5: accepts a new --threads option, Eric Blake, 2009/10/22
Re: [PATCH] md5: accepts a new --threads option, Jim Meyering, 2009/10/22
Re: [PATCH] md5: accepts a new --threads option, Paolo Bonzini, 2009/10/22
Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/23
Re: [PATCH] md5: accepts a new --threads option, Jim Meyering, 2009/10/23