|
From: | Chris Newton |
Subject: | [Duplicity-talk] Fwd: duplicity including files in backup that have not changed? |
Date: | Mon, 15 May 2017 14:57:30 -0700 |
Over the past several months I have slowly switched over about 600 machines from rdiff-backup to duplicity with an S3 back end at Soft Layer. Overall it has been great.
For some reason on four of my machines(all el5) incremental backups with duplicity sync about 1.2 million of 1.6 million files that I do not believe have changed. When I switch back to rdiff-backup the files do not get sync’d.
I can’t run duplicity currently as it uses about 8G of RAM and kills the machine, so I can’t do a verbose run to see exactly which files are changing. Would anyone have an idea as to a situation where duplicity would treat files as changed when rdiff-backup would not?
Another side effect of this is the memory footprint incremental backup duplicity process gets gigantic. I would expect it to store all files in the latest backup chain since the latest full, but it appears to read in all volumes from all chains? Here is a graph of the size of the duplicity memory footprint growing. Note full backups happened on the 1st/2nd and 15th/16th:
https://docs.google.com/
Thanks in advance.
Version info:
Version 0.7.11 released December 31, 2016
Python 2.7.9
The file list that both rdiff and duplicity use is the exact same file.
Rdiff backup command:
/usr/bin/rdiff-backup -b --print-statistics --tempdir /backups/tmp --remote-schema "/bin/nice -n 19 ssh -q -o StrictHostKeyChecking=no -C -o ServerAliveInterval=15 -o ServerAliveCountMax=30 -i ${IDENTITY_FILE} %s \"rdiff-backup --server\" || echo error: rdiff-backup failure" --include-globbing-filelist ${GLOBBING_FILELIST} --exclude '*' / address@hidden:
Duplicity backup command:
/opt/duplicity/bin/duplicity --include-filelist /etc/duplicity/conf/files-
Summary of some of the backup log:
Starting with rdiff:
Apr 7 00:20:14 zxy-app-01-p rdiff-backup: SourceFiles 1543774
Apr 7 00:20:14 zxy-app-01-p rdiff-backup: ChangedFiles 839
Apr 8 00:05:44 zxy-app-01-p rdiff-backup: SourceFiles 1548166
Apr 8 00:05:44 zxy-app-01-p rdiff-backup: ChangedFiles 11186
To duplicity:
Apr 11 06:11:16 zxy-app-01-p duplicity_backup: SourceFiles 1545039
Apr 11 06:11:16 zxy-app-01-p duplicity_backup: ChangedFiles 0
Apr 11 23:02:25 zxy-app-01-p duplicity_backup: SourceFiles 1545051
Apr 11 23:02:25 zxy-app-01-p duplicity_backup: ChangedFiles 1181203
Apr 13 01:34:44 zxy-app-01-p duplicity_backup: SourceFiles 1550345
Apr 13 01:34:44 zxy-app-01-p duplicity_backup: ChangedFiles 1181867
Apr 13 23:06:00 zxy-app-01-p duplicity_backup: SourceFiles 1550041
Apr 13 23:06:00 zxy-app-01-p duplicity_backup: ChangedFiles 1181203
Apr 14 23:14:08 zxy-app-01-p duplicity_backup: SourceFiles 1549683
Apr 14 23:14:08 zxy-app-01-p duplicity_backup: ChangedFiles 1181556
Back to rdiff backup:
Apr 29 10:56:00 zxy-app-01-p rdiff-backup: SourceFiles 1537895
Apr 29 10:56:00 zxy-app-01-p rdiff-backup: ChangedFiles 1
Apr 29 20:22:42 zxy-app-01-p rdiff-backup: SourceFiles 1537873
Apr 29 20:22:42 zxy-app-01-p rdiff-backup: ChangedFiles 89
Apr 30 20:23:21 zxy-app-01-p rdiff-backup: SourceFiles 1537850
Apr 30 20:23:21 zxy-app-01-p rdiff-backup: ChangedFiles 87
[Prev in Thread] | Current Thread | [Next in Thread] |