[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: better buffer size for copy

From: Robert Latham
Subject: Re: better buffer size for copy
Date: Sun, 20 Nov 2005 15:51:12 -0600
User-agent: Mutt/1.5.9i

On Sat, Nov 19, 2005 at 10:49:07AM -0500, Phillip Susi wrote:
> I don't see why the filesystem's cluster size should have a thing to do 
> with the buffer size used to copy files.  For optimal performance, the 
> larger the buffer, the better.  Diminishing returns applies of course, 
> so at some point the increase in buffer size results in little to no 
> further increase in performance, so that's the size you should use.  I 
> believe that the optimal size is about 64 KB. 

In local file systems, i'm sure you are correct.  If you are working
with a remote file system, however, the optimal size is on the order
of megabytes, not kilobytes.  For a specific example, consider the
PVFS2 file system, where the plateau in "blocksize vs. bandwitdh" is
two orders of magnitude larger than 64 KB.  PVFS2 is a parallel file
system for linux clusters.  I am not nearly as familiar with Lustre,
GPFS, or GFS, but I suspect those filesystems too would benefit from
block sizes larger than 64 KB.  

Are you taking umbrage at the idea of using st_blksize to direct how
large the transfer size should be for I/O?  I don't know what other
purpose st_blksize should have, nor are there any other fields which
are remotely valid for that purpose.  

Thanks for your feedback. 

Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Labs, IL USA                B29D F333 664A 4280 315B

reply via email to

[Prev in Thread] Current Thread [Next in Thread]