Re: [rdiff-backup-users] rdiff-backup performance -- slow operation on initial backup?

[Derek Atkins]

Just a quick update:  I tried making these changes on both sides and it
really didn't make a difference.  Full backup of 202852072 Kbytes
required 2267m25.913s (previously it took 2346m57.800s, so it only sped
up a factor of 3%.

Only thing I have not yet tried is running a raw rsync to see how fast
that runs.  I'll do that next.

So, back to my orignal question: anyone have any idea how to get initial
transfers to run faster (or indeed any significant data transfers)?

Thanks,

-derek

"Derek Atkins" <address@hidden> writes:

> Hi,
>
> I'm trying to use rdiff-backup to backup a bunch of servers.  One
> particular server contains about 160GB of data, but when I try to perform
> the rdiff-backup it's saving the data at a measly 1MB/s.
>
> Here's my configuration:
>
>   [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas]
>
> I ran a bunch of tests to try to figure out my bottlenecks.
>
> I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000) on the
> backup server.  Going directly to FreeNAS via NFS (bybassing encfs) I get
> 50.2MB/s.  If I run dd directly on the backup server (through encfs) I get
> 20.1MB/s.  If I go over SSH from the backup server to the target server
> and run the dd on the target server, then write to FreeNas through encfs
> declines to 7.6MB/s.
>
> Note that in those SSH tests, however, I forgot to turn off compression. 
> When I do that, the throughput for the dd test reduced to 6.6BM/s.  (Note
> that this is running simultaneously with a running rdiff-backup, so it's
> possible that they are reducing performance).
>
> Then I ran an scp test to the same target server; copying about 1.4GB of
> photos.  Files ranged in size from 10KB to 5MB.  When run in standard mode
> (displaying each file status) I got 4.4MB/s.  Running in quiet mode I get
> 5.1MB/s.
>
> So clearly the bottleneck is in rdiff-backup -- performance (IMHO) should
> not be significantly slower than the last dd-over-ssh test.  It appears
> rdiff-backup is slowing me down by a factor of 5x throughput versus scp.
>
> I found a message from Ben from 2005 where he suggests increasing the
> blocksize and conn_bufsiz settings in Globals.py:
> https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html
>
> What he didn't say was whether this needed to be changed on the target
> server, the backup server, or both.  Nor do I know if that would actually
> help this situation.
>
> Do you have any ideas?
>
> Thanks,
>
> -derek
>
> PS: According to rpm, both systems are running version 1.2.8.

-- 
       Derek Atkins                 617-623-3745
       address@hidden             www.ihtfp.com
       Computer and Internet Security Consultant