Re: [rdiff-backup-users] Performance

[Charles Duffy]

Greg Freemyer wrote:
> If you have millions of anything, then the filesystem selection
> probably matters more than the backup software.  Reiser3 is known to
> have been specifically tuned to handle huge numbers of small files.

Erk. I don't exactly want to start a flamewar here, but as someone who 
has repeatedly lost data under ReiserFS, let me point out that ext3 can 
be tuned a great deal to improve performance: Turn on htree support 
(which is specifically relevant for the lots-of-files case; in e2fsprogs 
this is the dir_index feature), turn up the commit timer (by default, 
all writes are committed every 5 seconds), set data=writeback (to reduce 
the guarantees made by the journaling subsystem to the level of those 
provided by JFS/XFS -- obviously only if you can more about performance 
than data integrity). If the files on your disk were written by a 
pre-2.6 kernel, move them off and back to get the much-improved orlov 
allocator reducing seek times.

A well-tuned ext3 may not quite be on par with ReiserFS when 
benchmarking against the strong points of the latter -- but it's much 
better than an untuned one, and still has fewer data integrity issues 
than ReiserFS.


All that said, it would be interesting to do some benchmarking to see 
where time is actually being spent -- waiting for network latency, 
waiting for the filesystem or doing things in userspace. oprofile should 
easily distinguish between the latter two cases; more primitive tools 
(comparing wall-clock/user/system execution, for instance) should 
provide more general data on the former.