Re: [rdiff-backup-users] how to shrink/purge/truncate the metadata file (and the file_statistics file)?

[Adrian Klaver]

On Sunday 03 January 2010 8:53:45 pm =JeffH wrote:
> Thanks Adrian.
>
>  > Are you sure that --remove-older-than is actually completing?
>
> yes.
>
>  > If there is more
>  > than one increment to be removed at a time and the --force switch is not
>  > used then the increments will not be removed.
>
> yes, and rdiff-backup immediately tells you this.
>
>  > You might want to do a
>  > rdiff-backup -l /backup/dir to see if in fact the increments have been
>  > removed.
>
> yes, the increments were removed. And your questions imply that the
> metadata and file_statistics files should indeed be shrink/purge/truncated.

Yes the files that no longer apply are pruned.


>
> Apologies for my somewhat lame original question. I'm not very familiar
> with the details of how rdiff-backup works under the hood.
>
>
> So, I removed yet more increments (all but the current mirror), did a
> --check-destination-dir, and then did another backup, which completed
> successfully.
>
> However, in retrospect, I was confused wrt the actual sizes of the metadata
> and file_statistics files -- upon further examination, they are orders of
> magnitude smaller than I'd thought: megabytes rather than gigabytes.
>
> I'm now wondering if part of what went wrong (my backup disk filled up
> unexpectedly) was that I'd made enough large changes on the source disk (I
> "shrank" a VM's virtual disk by 10s of gigs, plus I'd had a prior backup
> get a read error on a large .mp3 directory), that there essentially wasn't
> overall enough free space on the backup disk to house the both the VM's
> incrementally changed virtual disk files (10s of gigs) plus adding back
> into the mix all the previously-deleted .mp3 files (which, even tho they
> weren't present in the "current mirror", they /were/ present in prior
> "increments").

As you found the increments dir grows when the source files either shrink or 
expand. Since it is basically a versioning system it has to retain all the 
information for a file over the time frame since the last --remove-older-than 
operation. For the situation you mention, I am not sure how you would avoid the 
space issue without doing what you ended up doing.

>
> So this begs the question of how much headroom should the backup disk have
> in comparison to the source disk(s)?  My (current) backup disk is only
> somewhat larger than the amount of source data I have to backup.

One way to get a handle on your backup directory is to do:
rdiff-backup --list-increment-sizes /backup/dir
This will show you where your storage is being taken up. The amount you are 
going to need is going to be useage specific. The more increments and the 
bigger the incremental changes the more storage you are going to need. Remember 
increments work both ways, so shrinking the size of source files will increase 
the size of the incremental storage at least until they are pruned 
by --remove-older-than.

>
> After this, I'm thinking that it ought to be significantly larger.
>
> Is there a rule of thumb for how much larger the backup disk ought to be in
> comparison to the overall aggregate size of the backed up source data ?
>
> thanks,
>
> =JeffH
>
ki



-- 
Adrian Klaver
address@hidden