[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[rdiff-backup-users] known issues? trailing spaces on vfat, hyphen as ra

From: Marcel Cary
Subject: [rdiff-backup-users] known issues? trailing spaces on vfat, hyphen as range in regex, long filenames
Date: Fri, 1 Jun 2007 09:09:39 -0700 (PDT)

I'm trying to get rdiff-backup to replace my tar | gzip | split backup system that does full backups every time. So far I've found these issues:

1. rdiff-backup seems to choke on files with trailing spaces when backing
   up to a vfat filesystem

2. There appears to be a bug in the regex for quoting characters that
   causes '*' to not be quoted when it should

3. rdiff-backup turns some ~140 character filenames into ~260 filenames
   on vfat

I've seen that (3) is a known issue (bug 12823, fixed in CVS late 2005), but I would have expected that fix to make it into my version 1.0.4 (released early 2006). But I'm still having long filename issues.

Have other folks seen (1) and (2) behaviors? Are they known issues? I don't see them in the Savannah bug database. I'd be happy to file bugs if that would be helpful. Or perhaps I should be installing the latest stable rdiff-backup from source to avoid these issues.

The details:

My backup disk is currently formated with a FAT filesystem, which appears to disallow filenames with trailing spaces.

  $ sudo touch 'foo '
  touch: setting times of `foo ': No such file or directory
  $ sudo mkdir 'foo '
  mkdir: cannot create directory `foo ': Invalid argument
  $ mount
  /dev/sda1 on /media/usbdisk type vfat 

rdiff-backup chokes on a file with trailing spaces like this:

  Traceback (most recent call last):
    File "/usr/bin/rdiff-backup", line 23, in ?
    File "/usr/lib/python2.4/site-packages/rdiff_backup/Main.py", line 285, in 
    File "/usr/lib/python2.4/site-packages/rdiff_backup/Main.py", line 255, in 
      elif action == "backup": Backup(rps[0], rps[1])
    File "/usr/lib/python2.4/site-packages/rdiff_backup/Main.py", line 308, in 
      backup.Mirror(rpin, rpout)
    File "/usr/lib/python2.4/site-packages/rdiff_backup/backup.py", line 38, in 
      DestS.patch(dest_rpath, source_diffiter)
    File "/usr/lib/python2.4/site-packages/rdiff_backup/backup.py", line 218, 
in patch
      ITR(diff.index, diff)
    File "/usr/lib/python2.4/site-packages/rdiff_backup/rorpiter.py", line 288, 
in __call__
    File "/usr/lib/python2.4/site-packages/rdiff_backup/backup.py", line 548, 
in start_process
      if diff_rorp.isdir(): self.prepare_dir(diff_rorp, base_rp)
    File "/usr/lib/python2.4/site-packages/rdiff_backup/backup.py", line 574, 
in prepare_dir
    File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line 796, in 
  OSError: [Errno 22] Invalid argument: 
 ;070amily & ;067ommunity/;071ifts ;082eceived ;076ists '
  Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method 
GzipFile.__del__ of <gzip open file 
 mode 'wb' at 0xb7b35020 -0x486ef474>> ignored
  Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method 
GzipFile.__del__ of <gzip open file 
 mode 'wb' at 0xb7baff98 -0x486ef454>> ignored
  Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method 
GzipFile.__del__ of <gzip open file 
 mode 'wb' at 0xb7b35068 -0x486ef2b4>> ignored

It turns out that in this particular case, I can work around the problem by asking the owner of that directory to rename it; it was certainly an error to name the file that way.

In spite of the easy work-around, it might be worth fixing. I'd guess that escaping a trailing space would work, but might be difficult without escaping *all* spaces, which would make the transformed filenames uglier than necessary. (Note that a leading or interior space is no problem for vfat.) I'm sure there are other solutions as well.

Now for the regex error... after working around the previous issue, I get another stack trace on a file named /usr/share/guile/1.6/ice-9/and-let*.scm. Note the asterisk in the filename. Also, in the output from rdiff-backup:

  Characters needing quoting                   '^a-z0-9_ -.'

Note that the last dash is between a space and a period, which, in a regex character class, means from ord(" ") to ord("."). Asterisk falls within this range. A fix is to move the dash last in the character class to that it will not be interpretted as a range.

import re
r = re.compile("[^a-z0-9_ -.]|;")
r.sub("+", "and-let*.scm.2007-05-30T16:15:53-07:00.missing")
'and-let*.scm.2007-05-30+16+15+53-07+00.missing'  <-- asterisk not replaced
r = re.compile("[^a-z0-9_ .-]|;")
r.sub("+", "and-let*.scm.2007-05-30T16:15:53-07:00.missing")

And finally, the long filename issue. KDE's RSS reader Akregator stores feed data in files named by their URLs. For example:


But some URLs are longer. And feeds for search results have lots of '&' and '='. Between the escaping of those characters, the timestamp suffix added by rdiff-backup, and the escaping of the characters in the timestamp, several feeds with ~140 character URLs get transformed into
~260 character increment files.  And with that, rdiff-backup fails.

I ran across a link to duplicity, which, asside from encoding backups in a tar-style archive, appends metadata with a prepended directory. Maybe this would alleviate some of the long filename pain? For example, the increment for the above files could be named:


Note that many of the increments in the same directory share the same timestamp, so I would generally expect several diffs in each timestamp directory. The .diff.gz could also potentially be handled this way. Or maybe by generating random names as suggested in bug 12823 if that's done and working.

I'm currently working around the issue by not backing up those files:

--exclude-regexp '/home/[^/]+/\.kde/share/apps/akregator/Archive/[^/]{53,}'

I'm not ready to drop my old tarball strategy as long as rdiff-backup is skipping those files. Scarier still is that after adding the exclude regexp, I had to remove the entire backup repository to get a successful backup. I think that's because the first pass at some of those files did not require increments which append the 47 character (when escaped) timestamp suffix, and perhaps when excluding the files it was necessary to record their removal from the repository by creating an increment.

I'm thrilled to see that in spite of vfat's limitations, rdiff-backup can keep track of unix-type details like owner, group, permissions, and special files. I choose vfat so I can have (limited) access to backup data before I'm able to reinstall my Linux server after a failure.

I'm ready to get rid of all but the most recent tarball from my old strategy, and hopefully soon I can ditch the tarball strategy all together.

$ uname -a
Linux foo #1 Wed May 3 04:53:23 UTC 2006 i686 i686 i386 
$ python -V
Python 2.4.2
$ rdiff-backup --version
rdiff-backup 1.0.4


reply via email to

[Prev in Thread] Current Thread [Next in Thread]