[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [rdiff-backup-users] Memory usage during regressions
From: |
Claus-Justus Heine |
Subject: |
Re: [rdiff-backup-users] Memory usage during regressions |
Date: |
Sat, 06 Aug 2011 19:41:31 +0200 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20110706 Thunderbird/5.0 |
On 08/06/2011 07:13 PM, Robert Nichols wrote:
On 08/06/2011 08:35 AM, Claus-Justus Heine wrote:
[snip]
Then things start to slow down (swapping). It's a quite large backup
set, about
400G, with a large history. I doesn't seem to be a memory leak, as the
memory
usage stay at 3G. Just seems a little bit too much in principle.
Regression is concerned with only the two most recent sessions, so
Well, yes. So that piece of information was irrelevant, as rdiff-backups
keeps the most recent snapshot literally, and the older ones as deltas.
the amount of history should be irrelevant. What is the total
number of files being backed up and the size of the uncompressed
mirror_metadata snapshot?
zcat file_statistics.{latest_timestamp}.data.gz | tr '\0' '\n' | wc -l
zcat mirror_metadata.{latest_timestamp}.snapshot.gz | wc -c
That would be more indicative of the amount of data that needs to be
Quite a bit of data:
address@hidden rdiff-backup-data $\
zcat file_statistics.2011-07-24T20:02:14+02:00.data.gz|\
tr '\0' '\n' |\
wc -l
2698977
address@hidden rdiff-backup-data $\
zcat mirror_metadata.2011-07-30T05\:56\:36+02\:00.snapshot.gz |\
wc -c
732509974
So roughly 2.5 millon files, metadata is about 700M.
kept in memory during the regression. FWIW, I'm seeing memory usage
of about 480MB during regression of a backup of about 250,000 files,
Well mine is a factor of 10 larger, roughly (good that memory
consumption does not scale linearily ... ;)
Still, this seems to be insane. Reading this month's archives (and given
that the last release of rdiff-backup was about '09) it seems that I
would have to fix that by myself, it seems. Or live with it. Or buy a
backup server with more memory ...
Of course, in principle the core-size of rdiff-backup is not a problem,
on a decent OS the part of the core currently not in use would just be
swapped out. There is only a problem if the program constantly traverses
the allocated buffers. This is what I suspect. It takes ages (2 or 3
days) to finally finish the regression.
Thanks for your quick response!
Best,
Claus