[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Duplicity-talk] Backup succeeding but not writing manifest / sigtar

From: Mark Grandi
Subject: Re: [Duplicity-talk] Backup succeeding but not writing manifest / sigtar
Date: Mon, 29 Feb 2016 21:50:28 -0700

I am backing up root, because on Mac OS X, almost all of the applications are stored in /Applications , (shared between all users), so I am backing up root, and excluding all of the other directories besides /Users , but that doesn't seem to stop Duplicity from trying to access them anyway, and failing due to not having the read permissions.

Anyway, I looked into my problem myself since I had a free afternoon, and found myself honestly quite concerned with what some of the code does, or rather doesn't do. For instance, with Par2Backend, if for some reason the par2 command doesn't return 0, then it just silently doesn't create any par2 files without any notice to the user. In LocalBackend, if any of the methods in there that have try/except blocks, in the except blocks, they are either 'pass' or just 'return False', and often the caller doesn't care about the return result anyway, so any errors that occur during moving/copying files are silently discarded and not presented to the user, nor does it invoke any of the retry logic in Duplicity.

I mentioned the two above things, because I had trouble compiling the par2 binary on my Mac, and upon googling, I found a Mac OS X app that is a GUI frontend to par and par2, and it appears that the author created their own fork of par2, that works on mac (and has improvements such as using grand central dispatch, the website is here: https://gp.home.xs4all.nl/Site/MacPAR_deLuxe.html). Anyway, it sees this version of par2 does not like symlinks, and the Par2Backend class uses symlinks when creating the par2 files. Discovering why the par2 files were not being created required me stepping into the debugger, because no errors are logged to the user.

Once I changed the symlink to a hardlink, and modified the command line being passed to par2 somewhat, I was stuck on the issue that I originally posted:  the difftar files + par2 files were being moved to the correct spot, but not the manifest or sigtar files and their par2 counterparts. Stepping into the debugger, i found that the difftar files are being written with Par2Backend wrapping LocalBackend.put(), but the sigtar and manifest files are wrapping LocalBackend._move(). However, LocalBackend._move() is using Path.rename() instead of Path.move() ! It even says in the docstring for Path.move(): "Like rename but destination may be on different file system". 

So there lies my problem, my backup destination was on a different file system, but due to LocalBackend._move() calling the wrong underlying Path method, and swallowing exceptions and only returning false (that Par2Backend doesn't even check), it was not moving the sigtar and manifest files + par2 files because of "[Errno 18] Cross-device link". 

The patch to fix this is just simply replacing LocalBackend._move() with:

    def _move(self, source_path, remote_filename):
        target_path = self.remote_pathdir.append(remote_filename)
            source_path.setdata() # maybe only needed in Par2Backend?
            return True
        except OSError as e:
            return False

This does not solve the concerns I mentioned with swallowing exceptions however. 


On Feb 25, 2016, at 9:37 AM, Kenneth Loafman <address@hidden> wrote:

If you are backing up the system directories use "sudo".

The errors should not have caused a failure.  Try again without the par+ schema, just file://.

On Thu, Feb 25, 2016 at 2:40 AM, Mark Grandi <address@hidden> wrote:
Running the command below seems to work, the backup completes fine, and the errors seem to just correspond to permission errors (i am excluding those directories but i guess duplicity still tries to read inside them....), but when the backup finishes, it doesn't actually write the sigtar or manifest files to the backup location, so essentially the backup fails for all duplicity cares. Running 'collection-status' will delete the sigtar and manifest files from the archive_dir directory, because they are not present in the backup location, and 'cleanup' will just list all the files (the gpg and par2 files) as 'extra files', for the same reason. Why is duplicity not writing / copying these files to the backup directory? 

It seems that it creates the  manifest.gpg and sigtar.gpg files and writes them to the archive_dir, but fails to copy them over to the backup location. If I copy these files to the backup location manually , then it suddenly recognizes it as a backup. 


duplicity --encrypt-sign-key 0x2CB2C0948891C4830B11AFB6E7B785F6FA5CBE16 --log-file ~/Temp/test_duplicity.log --name "Gypaetus-mgrandi" --use-agent --exclude-device-files --volsize 1024 --file-prefix "Gypaetus-mgrandi-" --exclude-filelist /Users/markgrandi/duplicity_exclude.txt / par2+file:///Volumes/mgrandi_256SSD/backup

log entries:

Reading globbing filelist /Users/markgrandi/duplicity_exclude.txt
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
Error accessing possibly locked file /.DocumentRevisions-V100
Error accessing possibly locked file /.Spotlight-V100
Error accessing possibly locked file /.Trashes
Error accessing possibly locked file /.file
Error accessing possibly locked file /.fseventsd
Error accessing possibly locked file /Users/Guest/.CFUserTextEncoding
Error accessing possibly locked file /Users/Guest/Desktop
Error accessing possibly locked file /Users/Guest/Documents
Error accessing possibly locked file /Users/Guest/Downloads
Error accessing possibly locked file /Users/Guest/Library
Error accessing possibly locked file /Users/Guest/Movies
Error accessing possibly locked file /Users/Guest/Music
Error accessing possibly locked file /Users/Guest/Pictures
Error accessing possibly locked file /Users/Guest/Public/Drop Box
Error accessing possibly locked file /Users/markgrandi/.viminfo
Error accessing possibly locked file /Users/markgrandi/Library/Saved Application State/com.adobe.flashplayer.installmanager.savedState
Warning, found incomplete backup sets, probably left from aborted session
--------------[ Backup Statistics ]--------------
StartTime 1456382712.45 (Wed Feb 24 23:45:12 2016)
EndTime 1456386174.70 (Thu Feb 25 00:42:54 2016)
ElapsedTime 3462.25 (57 minutes 42.25 seconds)
SourceFiles 429045
SourceFileSize 22295414551 (20.8 GB)
NewFiles 429045
NewFileSize 22295414551 (20.8 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 429045
RawDeltaSize 22276747699 (20.7 GB)
TotalDestinationSizeChange 13905691762 (13.0 GB)
Errors 16

contents of ~/.cache/<archive_dir>/:

Gypaetus:Gypaetus-mgrandi markgrandi$ ls -la
total 1250080
drwxr-xr-x  6 markgrandi  staff        204 Feb 25 00:45 .
drwxr-xr-x  5 markgrandi  staff        170 Feb 24 23:45 ..
-rw-------  1 markgrandi  staff  295425508 Feb 25 00:44 Gypaetus-mgrandi-duplicity-full-signatures.20160225T064512Z.sigtar.gpg
-rw-------  1 markgrandi  staff  294998550 Feb 25 00:45 Gypaetus-mgrandi-duplicity-full-signatures.20160225T064512Z.sigtar.gz
-rw-------  1 markgrandi  staff   46875395 Feb 25 00:45 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.manifest
-rw-------  1 markgrandi  staff    2731077 Feb 25 00:45 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.manifest.gpg

contents of the backup dir:

Gypaetus:backup markgrandi$ ls -la
total 29885864
drwxr-xr-x  41 markgrandi  staff        1394 Feb 25 00:44 .
drwxrwxr-x  19 markgrandi  staff         714 Feb 24 23:42 ..
-rw-------   1 markgrandi  staff  1073732207 Feb 24 23:50 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol1.difftar.gpg
-rw-------   1 markgrandi  staff       40456 Feb 24 23:50 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol1.difftar.gpg.par2
-rw-------   1 markgrandi  staff   107736576 Feb 24 23:50 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol1.difftar.gpg.vol000+200.par2
-rw-------   1 markgrandi  staff  1073782430 Feb 25 00:32 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol10.difftar.gpg
-rw-------   1 markgrandi  staff       40456 Feb 25 00:32 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol10.difftar.gpg.par2
-rw-------   1 markgrandi  staff   107741376 Feb 25 00:32 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol10.difftar.gpg.vol000+200.par2
-rw-------   1 markgrandi  staff  1073806871 Feb 25 00:36 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol11.difftar.gpg
-rw-------   1 markgrandi  staff       40456 Feb 25 00:36 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol11.difftar.gpg.par2
-rw-------   1 markgrandi  staff   107743776 Feb 25 00:36 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol11.difftar.gpg.vol000+200.par2
-rw-------   1 markgrandi  staff  1073771432 Feb 25 00:40 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol12.difftar.gpg
-rw-------   1 markgrandi  staff       40456 Feb 25 00:40 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol12.difftar.gpg.par2
-rw-------   1 markgrandi  staff   107740576 Feb 25 00:40 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol12.difftar.gpg.vol000+200.par2
-rw-------   1 markgrandi  staff  1020263590 Feb 25 00:44 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol13.difftar.gpg
-rw-------   1 markgrandi  staff       40456 Feb 25 00:44 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol13.difftar.gpg.par2
-rw-------   1 markgrandi  staff   102387776 Feb 25 00:44 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol13.difftar.gpg.vol000+200.par2
-rw-------   1 markgrandi  staff  1073752418 Feb 24 23:57 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol2.difftar.gpg
-rw-------   1 markgrandi  staff       40456 Feb 24 23:57 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol2.difftar.gpg.par2
-rw-------   1 markgrandi  staff   107738976 Feb 24 23:57 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol2.difftar.gpg.vol000+200.par2
-rw-------   1 markgrandi  staff  1073807227 Feb 25 00:03 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol3.difftar.gpg
-rw-------   1 markgrandi  staff       40456 Feb 25 00:03 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol3.difftar.gpg.par2
-rw-------   1 markgrandi  staff   107743776 Feb 25 00:03 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol3.difftar.gpg.vol000+200.par2
-rw-------   1 markgrandi  staff  1073807149 Feb 25 00:09 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol4.difftar.gpg
-rw-------   1 markgrandi  staff       40456 Feb 25 00:09 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol4.difftar.gpg.par2
-rw-------   1 markgrandi  staff   107743776 Feb 25 00:09 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol4.difftar.gpg.vol000+200.par2
-rw-------   1 markgrandi  staff  1073742782 Feb 25 00:12 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol5.difftar.gpg
-rw-------   1 markgrandi  staff       40456 Feb 25 00:12 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol5.difftar.gpg.par2
-rw-------   1 markgrandi  staff   107737376 Feb 25 00:12 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol5.difftar.gpg.vol000+200.par2
-rw-------   1 markgrandi  staff  1073808088 Feb 25 00:20 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol6.difftar.gpg
-rw-------   1 markgrandi  staff       40456 Feb 25 00:20 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol6.difftar.gpg.par2
-rw-------   1 markgrandi  staff   107744576 Feb 25 00:20 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol6.difftar.gpg.vol000+200.par2
-rw-------   1 markgrandi  staff  1073817767 Feb 25 00:23 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol7.difftar.gpg
-rw-------   1 markgrandi  staff       40456 Feb 25 00:23 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol7.difftar.gpg.par2
-rw-------   1 markgrandi  staff   107745376 Feb 25 00:23 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol7.difftar.gpg.vol000+200.par2
-rw-------   1 markgrandi  staff  1073811095 Feb 25 00:25 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol8.difftar.gpg
-rw-------   1 markgrandi  staff       40456 Feb 25 00:25 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol8.difftar.gpg.par2
-rw-------   1 markgrandi  staff   107744576 Feb 25 00:25 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol8.difftar.gpg.vol000+200.par2
-rw-------   1 markgrandi  staff  1073788706 Feb 25 00:28 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol9.difftar.gpg
-rw-------   1 markgrandi  staff       40456 Feb 25 00:28 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol9.difftar.gpg.par2
-rw-------   1 markgrandi  staff   107742176 Feb 25 00:28 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol9.difftar.gpg.vol000+200.par2


Duplicity-talk mailing list

Duplicity-talk mailing list

Attachment: signature.asc
Description: Message signed with OpenPGP using GPGMail

reply via email to

[Prev in Thread] Current Thread [Next in Thread]