duplicity-talk
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Duplicity-talk] Re: AssertionError: snapshot: Using rdiffdir as an rsyn


From: Mike Mestnik
Subject: [Duplicity-talk] Re: AssertionError: snapshot: Using rdiffdir as an rsync replacement.
Date: Fri, 29 Aug 2008 10:25:14 -0500
User-agent: Mutt/1.4.2.1i

I looked at some of the code, I'd vary much like a patch for this.
Looks like if I knew python better I could easily skip over the
effected files with an error.  I'd like a patch with two goals in mind:

1. Getting the operation to complete, regardless of consequenses.
      I can understand why this feature would never get checked in.
2. Increased reporting about the problem.
      This Assertion is great it prevented my operation from being
      completed, but the formatt should be changed and this change
      should be included in the next release.

Example format:
AssertionError: <filename>: snapshot

However I'd like the Assertion removed, I can stand to have
one/some/most/all files not being replicated.  What I'd like to see is
a program that can itterate over my entire dataset without crashing in
some way.  If it was easy I'd have done it.

On Tue, Aug 26, 2008 at 11:17:37AM -0500, Mike Mestnik wrote:
> Using rdiffdir we were able to generate signatures once per every
> several replicating runs.  This gave an obvious advantage over rsync
> and also allowed us to get a much smaller changeset, that rsync was
> making difficult(sshing to localhost with no eyncription proved to be
> the solution).
> 
> The backend storage is a netapp and we are using the snapshots to do
> the archiving part of our backups.  Having the smaller changesets
> generated by rdiffdir applied to the NFS mounted filesystem allowed
> the netapp to make better use of it's avliable storage.
> 
> Unfortunatly none of the tools we have tried has been succsessfull at
> replicating our dataset.  It's many folders of many small to large
> files that have accumulated over a decade, many hundreds of GB of user
> data.  rsync versions prior to 3 run out of memory, rsync 3 mostly
> worked.
> 
> Could you identify what these errors mean?
> 
> AssertionError: snapshot
> 
> ----- Forwarded message from Cron Daemon -----
> 
> > From: root (Cron Daemon)
> > To: root
> > Subject: [Alert-cron] Cron <root> /usr/local/bin/lockrun
> >     --lockfile=/tmp/ad3.lockrun -- /usr/local/bin/lockrun
> >     --lockfile=/tmp/master1.lockrun --wait --
> >     /etc/scripts/rsync.pl ad3 >> /var/log/rsync-ad3.log
> > Date: Tue, 26 Aug 2008 02:07:09 -0500 (CDT)
> > 
> > 1219734425 system( cd /sqldump; cd /sqldump/.snapshot/nightly.0; 
> > /usr/local/bin/rdiffdir --exclude-other-filesystems 
> > --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' delta 
> > /rsync/61301/fetket/vol6/sqldump.sig . - | /usr/local/bin/rdiffdir 
> > --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> > --exclude-regexp='.*Maildir/..*' patch /rsync/61301/fetket/vol6/sqldump - 
> > >> /home/rsync/ad3:sqldump; );
> > cd: can't cd to /sqldump/.snapshot/nightly.0
> > Traceback (most recent call last):
> >   File "/usr/local/bin/rdiffdir", line 193, in ?
> >     if __name__ == "__main__": main()
> >   File "/usr/local/bin/rdiffdir", line 184, in main
> >     patch(file_args[0], get_fileobj(file_args[1], "rb"))
> >   File "/usr/local/bin/rdiffdir", line 159, in patch
> >     patchdir.Patch(path.Path(dirname), deltafp)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 33, in Patch
> >     patch_diff_tarfile(base_path, diff_tarfile)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 60, in patch_diff_tarfile
> >     ITR(basis_path.index, basis_path, diff_ropath)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/lazy.py", line 
> > 297, in __call__
> >     last_branch.fast_process, args)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/robust.py", line 
> > 31, in check_common_error
> >     try: return function(*args)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 257, in fast_process
> >     assert diff_ropath.difftype == "diff", diff_ropath.difftype
> > AssertionError: snapshot
> > Traceback (most recent call last):
> >   File "/usr/local/bin/rdiffdir", line 193, in ?
> >     if __name__ == "__main__": main()
> >   File "/usr/local/bin/rdiffdir", line 182, in main
> >     else: write_delta(file_args[1], sig_infp, delta_outfp)
> >   File "/usr/local/bin/rdiffdir", line 147, in write_delta
> >     diffdir.write_block_iter(delta_iter, outfp)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py", line 
> > 530, in write_block_iter
> >     for block in block_iter: fp.write(block.data)
> > IOError: [Errno 32] Broken pipe
> > 
> > _______________________________________________
> > Alert-cron mailing list
> > address@hidden
> > http://weebles.visi.com/mailman/listinfo/alert-cron
> 
> ----- End forwarded message -----
> ----- Forwarded message from Cron Daemon <root> -----
> 
> > From: root (Cron Daemon)
> > To: root
> > Subject: [Alert-cron] Cron <root> /usr/local/bin/lockrun
> >     --lockfile=/tmp/apache.lockrun -- /usr/local/bin/lockrun
> >     --lockfile=/tmp/master2.lockrun --wait --
> >     /etc/scripts/rsync.pl apache >> /var/log/rsync-apache.log
> > Date: Tue, 26 Aug 2008 02:25:42 -0500 (CDT)
> > 
> > 1219734490 system( cd /rsync/61327/jeeves/apache/www; cd 
> > /rsync/61327/jeeves/apache/www/.snapshot/nightly.0; /usr/local/bin/rdiffdir 
> > --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> > --exclude-regexp='.*Maildir/..*' delta /rsync/61327/f760-2/vol0/www.sig . - 
> > | /usr/local/bin/rdiffdir --exclude-other-filesystems 
> > --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' patch 
> > /rsync/61327/f760-2/vol0/www - >> /home/rsync/apache:www; );
> > Traceback (most recent call last):
> >   File "/usr/local/bin/rdiffdir", line 193, in ?
> >     if __name__ == "__main__": main()
> >   File "/usr/local/bin/rdiffdir", line 184, in main
> >     patch(file_args[0], get_fileobj(file_args[1], "rb"))
> >   File "/usr/local/bin/rdiffdir", line 159, in patch
> >     patchdir.Patch(path.Path(dirname), deltafp)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 33, in Patch
> >     patch_diff_tarfile(base_path, diff_tarfile)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 60, in patch_diff_tarfile
> >     ITR(basis_path.index, basis_path, diff_ropath)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/lazy.py", line 
> > 297, in __call__
> >     last_branch.fast_process, args)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/robust.py", line 
> > 31, in check_common_error
> >     try: return function(*args)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 257, in fast_process
> >     assert diff_ropath.difftype == "diff", diff_ropath.difftype
> > AssertionError: snapshot
> > Traceback (most recent call last):
> >   File "/usr/local/bin/rdiffdir", line 193, in ?
> >     if __name__ == "__main__": main()
> >   File "/usr/local/bin/rdiffdir", line 182, in main
> >     else: write_delta(file_args[1], sig_infp, delta_outfp)
> >   File "/usr/local/bin/rdiffdir", line 147, in write_delta
> >     diffdir.write_block_iter(delta_iter, outfp)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py", line 
> > 530, in write_block_iter
> >     for block in block_iter: fp.write(block.data)
> > IOError: [Errno 32] Broken pipe
> > 1219734507 system( cd /rsync/61327/jeeves/apache/ftp; cd 
> > /rsync/61327/jeeves/apache/ftp/.snapshot/nightly.0; /usr/local/bin/rdiffdir 
> > --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> > --exclude-regexp='.*Maildir/..*' delta /rsync/61327/f760-2/vol0/ftp.sig . - 
> > | /usr/local/bin/rdiffdir --exclude-other-filesystems 
> > --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' patch 
> > /rsync/61327/f760-2/vol0/ftp - >> /home/rsync/apache:ftp; );
> > Traceback (most recent call last):
> >   File "/usr/local/bin/rdiffdir", line 193, in ?
> >     if __name__ == "__main__": main()
> >   File "/usr/local/bin/rdiffdir", line 184, in main
> >     patch(file_args[0], get_fileobj(file_args[1], "rb"))
> >   File "/usr/local/bin/rdiffdir", line 159, in patch
> >     patchdir.Patch(path.Path(dirname), deltafp)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 33, in Patch
> >     patch_diff_tarfile(base_path, diff_tarfile)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 60, in patch_diff_tarfile
> >     ITR(basis_path.index, basis_path, diff_ropath)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/lazy.py", line 
> > 297, in __call__
> >     last_branch.fast_process, args)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/robust.py", line 
> > 31, in check_common_error
> >     try: return function(*args)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 257, in fast_process
> >     assert diff_ropath.difftype == "diff", diff_ropath.difftype
> > AssertionError: snapshot
> > Traceback (most recent call last):
> >   File "/usr/local/bin/rdiffdir", line 193, in ?
> >     if __name__ == "__main__": main()
> >   File "/usr/local/bin/rdiffdir", line 182, in main
> >     else: write_delta(file_args[1], sig_infp, delta_outfp)
> >   File "/usr/local/bin/rdiffdir", line 147, in write_delta
> >     diffdir.write_block_iter(delta_iter, outfp)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py", line 
> > 530, in write_block_iter
> >     for block in block_iter: fp.write(block.data)
> > IOError: [Errno 32] Broken pipe
> > 1219734536 system( cd /rsync/61327/jeeves/apache/clockwork; cd 
> > /rsync/61327/jeeves/apache/clockwork/.snapshot/nightly.0; 
> > /usr/local/bin/rdiffdir --exclude-other-filesystems 
> > --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' delta 
> > /rsync/61327/f760-2/vol0/clockwork.sig . - | /usr/local/bin/rdiffdir 
> > --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> > --exclude-regexp='.*Maildir/..*' patch /rsync/61327/f760-2/vol0/clockwork - 
> > >> /home/rsync/apache:clockwork; );
> > Traceback (most recent call last):
> >   File "/usr/local/bin/rdiffdir", line 193, in ?
> >     if __name__ == "__main__": main()
> >   File "/usr/local/bin/rdiffdir", line 184, in main
> >     patch(file_args[0], get_fileobj(file_args[1], "rb"))
> >   File "/usr/local/bin/rdiffdir", line 159, in patch
> >     patchdir.Patch(path.Path(dirname), deltafp)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 33, in Patch
> >     patch_diff_tarfile(base_path, diff_tarfile)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 63, in patch_diff_tarfile
> >     ITR(diff_ropath.index, basis_path, diff_ropath)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/lazy.py", line 
> > 297, in __call__
> >     last_branch.fast_process, args)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/robust.py", line 
> > 31, in check_common_error
> >     try: return function(*args)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 248, in fast_process
> >     diff_ropath.copy(self.base_path.new_index(index))
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/path.py", line 
> > 353, in copy
> >     if self.isreg(): other.writefileobj(self.open("rb"))
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/path.py", line 
> > 492, in writefileobj
> >     fout = self.open("wb")
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/path.py", line 
> > 444, in open
> >     else: result = open(self.name, mode)
> > IOError: [Errno 28] No space left on device: 
> > '/rsync/61327/f760-2/vol0/clockwork/bigip-apache/conf/access.conf-dist'
> > Traceback (most recent call last):
> >   File "/usr/local/bin/rdiffdir", line 193, in ?
> >     if __name__ == "__main__": main()
> >   File "/usr/local/bin/rdiffdir", line 182, in main
> >     else: write_delta(file_args[1], sig_infp, delta_outfp)
> >   File "/usr/local/bin/rdiffdir", line 147, in write_delta
> >     diffdir.write_block_iter(delta_iter, outfp)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py", line 
> > 530, in write_block_iter
> >     for block in block_iter: fp.write(block.data)
> > IOError: [Errno 32] Broken pipe
> > 1219734645 system( cd /rsync/61327/jeeves/apache/cvs; cd 
> > /rsync/61327/jeeves/apache/cvs/.snapshot/nightly.0; /usr/local/bin/rdiffdir 
> > --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> > --exclude-regexp='.*Maildir/..*' delta /rsync/61327/f760-2/vol0/cvs.sig . - 
> > | /usr/local/bin/rdiffdir --exclude-other-filesystems 
> > --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' patch 
> > /rsync/61327/f760-2/vol0/cvs - >> /home/rsync/apache:cvs; );
> > 1219734718 system( cd /rsync/61327/jeeves/apache/tftpboot; cd 
> > /rsync/61327/jeeves/apache/tftpboot/.snapshot/nightly.0; 
> > /usr/local/bin/rdiffdir --exclude-other-filesystems 
> > --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' delta 
> > /rsync/61327/f760-2/vol0/tftpboot.sig . - | /usr/local/bin/rdiffdir 
> > --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> > --exclude-regexp='.*Maildir/..*' patch /rsync/61327/f760-2/vol0/tftpboot - 
> > >> /home/rsync/apache:tftpboot; );
> > Traceback (most recent call last):
> >   File "/usr/local/bin/rdiffdir", line 193, in ?
> >     if __name__ == "__main__": main()
> >   File "/usr/local/bin/rdiffdir", line 184, in main
> >     patch(file_args[0], get_fileobj(file_args[1], "rb"))
> >   File "/usr/local/bin/rdiffdir", line 159, in patch
> >     patchdir.Patch(path.Path(dirname), deltafp)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 33, in Patch
> >     patch_diff_tarfile(base_path, diff_tarfile)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 60, in patch_diff_tarfile
> >     ITR(basis_path.index, basis_path, diff_ropath)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/lazy.py", line 
> > 297, in __call__
> >     last_branch.fast_process, args)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/robust.py", line 
> > 31, in check_common_error
> >     try: return function(*args)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 257, in fast_process
> >     assert diff_ropath.difftype == "diff", diff_ropath.difftype
> > AssertionError: snapshot
> > Traceback (most recent call last):
> >   File "/usr/local/bin/rdiffdir", line 193, in ?
> >     if __name__ == "__main__": main()
> >   File "/usr/local/bin/rdiffdir", line 182, in main
> >     else: write_delta(file_args[1], sig_infp, delta_outfp)
> >   File "/usr/local/bin/rdiffdir", line 147, in write_delta
> >     diffdir.write_block_iter(delta_iter, outfp)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py", line 
> > 530, in write_block_iter
> >     for block in block_iter: fp.write(block.data)
> > IOError: [Errno 32] Broken pipe
> > 1219734721 system( cd /rsync/61327/jeeves/apache/staff; cd 
> > /rsync/61327/jeeves/apache/staff/.snapshot/nightly.0; 
> > /usr/local/bin/rdiffdir --exclude-other-filesystems 
> > --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' delta 
> > /rsync/61327/f760-2/vol0/staff.sig . - | /usr/local/bin/rdiffdir 
> > --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> > --exclude-regexp='.*Maildir/..*' patch /rsync/61327/f760-2/vol0/staff - >> 
> > /home/rsync/apache:staff; );
> > Traceback (most recent call last):
> >   File "/usr/local/bin/rdiffdir", line 193, in ?
> >     if __name__ == "__main__": main()
> >   File "/usr/local/bin/rdiffdir", line 184, in main
> >     patch(file_args[0], get_fileobj(file_args[1], "rb"))
> >   File "/usr/local/bin/rdiffdir", line 159, in patch
> >     patchdir.Patch(path.Path(dirname), deltafp)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 33, in Patch
> >     patch_diff_tarfile(base_path, diff_tarfile)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 60, in patch_diff_tarfile
> >     ITR(basis_path.index, basis_path, diff_ropath)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/lazy.py", line 
> > 297, in __call__
> >     last_branch.fast_process, args)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/robust.py", line 
> > 31, in check_common_error
> >     try: return function(*args)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> > 257, in fast_process
> >     assert diff_ropath.difftype == "diff", diff_ropath.difftype
> > AssertionError: snapshot
> > Traceback (most recent call last):
> >   File "/usr/local/bin/rdiffdir", line 193, in ?
> >     if __name__ == "__main__": main()
> >   File "/usr/local/bin/rdiffdir", line 182, in main
> >     else: write_delta(file_args[1], sig_infp, delta_outfp)
> >   File "/usr/local/bin/rdiffdir", line 147, in write_delta
> >     diffdir.write_block_iter(delta_iter, outfp)
> >   File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py", line 
> > 530, in write_block_iter
> >     for block in block_iter: fp.write(block.data)
> > IOError: [Errno 32] Broken pipe
> > 1219734723 system( cd /rsync/61327/jeeves/apache/bitstream; cd 
> > /rsync/61327/jeeves/apache/bitstream/.snapshot/nightly.0; 
> > /usr/local/bin/rdiffdir --exclude-other-filesystems 
> > --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' delta 
> > /rsync/61327/f760-2/vol0/bitstream.sig . - | /usr/local/bin/rdiffdir 
> > --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> > --exclude-regexp='.*Maildir/..*' patch /rsync/61327/f760-2/vol0/bitstream - 
> > >> /home/rsync/apache:bitstream; );
> > 
> > _______________________________________________
> > Alert-cron mailing list
> > address@hidden
> > http://weebles.visi.com/mailman/listinfo/alert-cron
> 
> ----- End forwarded message -----
> 
> -- 
> /****************************************************************
>  *   Mike Mestnik: Junior Admin          612-395-8932           *
>  *      address@hidden                  VISI                 *
>  ****************************************************************/
>  Alt address: address@hidden

-- 
/****************************************************************
 *   Mike Mestnik: Junior Admin          612-395-8932           *
 *      address@hidden                  VISI                 *
 ****************************************************************/
 Alt address: address@hidden




reply via email to

[Prev in Thread] Current Thread [Next in Thread]