[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [rdiff-backup-users] Memory benchmarks and some questions

From: David
Subject: Re: [rdiff-backup-users] Memory benchmarks and some questions
Date: Fri, 9 May 2008 20:07:14 +0200

> Hi, not sure what you mean by memory limited

Well, by my calculations (500 kb on average per 10,000 files),
rdiff-backup would need 64 MB for 1.3 million files, and 1 GB for 20
million files. That could be a problem. I haven't actually tested with
that many files yet.

Also, my tests were for all the files in a single directory. It's
possible that rdiff-backup uses less memory when the files are spread
over a directory structure.

> , but I definitely back up lots
> of repos with rdiff-backup and have never noticed any problems with memory
> usage.

What is the largest number of files you backup at once with
rdiff-backup? Could you check how much RAM rdiff-backup uses for that

> However transferring the same files with rsync takes much more RAM,
> as rsync loads the whole file list in memory, rdiff-backup doesn't seem to.

New versions of rsync (after 3.0.0) use an incremental file list, so
it doesn't need to keep the entire file list in memory.

> Perhaps there is some ceiling you are not reaching in terms of counting
> files before more ram consumption.
> Just my 2c.

With my tests I got up to around 60,000 files and the memory used was
still increasing with each new set of 10,000. So there's no reason to
think it wouldn't keep growing.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]