Re: Feature request for md5sum

[Scott Nichol]

Thanks.  The existence of stdbuf had escaped my attention; to the extent 
it works, it should do the trick nicely.

Scott Nichol

----- Original Message ----- 
From: "Pádraig Brady" <address@hidden>
To: "Scott Nichol" <address@hidden>
Cc: <address@hidden>
Sent: Friday, February 19, 2010 5:41 PM
Subject: Re: Feature request for md5sum


On 19/02/10 17:55, Scott Nichol wrote:
> I use md5sum a lot, often on network-filesystem-accessed files so that
> readers can be sure they are reading files after writers have 
> completed.
> The performance of md5sum is generally acceptable, but for large
> (gigabyte+) files even on a fast (1 Gbps) LAN, the performance would 
> be
> considerably better if I could specify on the command line a buffer 
> size
> to use.  I imagine the md5sum code using this value in a setvbuf call.
> I use this technique (setting larger buffer sizes for network-accessed
> files) frequently and know md5sum and other utilities would benefit 
> from
> it.

Yes they would benefit, however it would be nice
not to push this complexity into the tools.

Note the buffer size has changed in v8.1 from 4KiB to 32KiB,
which might help.

Also since v7.5 there is a stdbuf utility available
which can be used to change the buffer size for stdin at least:
   stdbuf -i1M md5sum ...
Note there were issues with glibc when I was testing that,
so don't be surprised if the setting ignored.

Note the linux nfs client should be doing adaptive
read ahead to abstract you from this latency issue.
Check the rsize nfs setting as that may limit the
size of read ahead done.

cheers,
Pádraig.