[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[rdiff-backup-users] PATCHES for Cross-Platform-Compatibility (finally)
|
From: |
sirafil |
|
Subject: |
[rdiff-backup-users] PATCHES for Cross-Platform-Compatibility (finally) |
|
Date: |
Fri, 28 Nov 2008 13:04:48 +0100 |
|
User-agent: |
Thunderbird 2.0.0.18 (Windows/20081105) |
Hi *,
sorry to the mailinglist for the cryptic mangled patches.
In my intial posting about 'Feature Suggestions for
Cross-Platform-Compatibility'
I wrote about some feature enhacements (I think) would benefit rdiff-backup
while
keeping full backward compatibility:
http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rdiff-backup-23/feature-suggestions-for-cross-platform-compatibility-93227/
Now this functionality is implemented and (well-)tested locally on Windows XP
NTFS
and Linux EXT2+EXT3.
During learning Python and reviewing the rdiff-backup code I fixed two small
bugs
in unrelated modules and cleaned up/updated the man-page.
BUGS Fixed:
- module eas_acls.py, Line: 224: comment position was erroneously calculated
before,
resulting in a (potential) nasty bug, truncating AccessControlLists-files
- module fs_abilities.py, get_ctq_from_fsas():
WAS pot. bug before, because unconditionally only ';' was quoted,
even if overridden with another character
Here are the patches, first the modifications to the man/help-page to explain
most of the reasons and added functionality:
[code]
--- rdiff-backup/rdiff-backup.1 2008-08-20 02:37:41.000000000 +0000
+++ rdiff-backup-mycode/rdiff-backup.1 2008-11-09 11:40:58.000000000 +0000
@@ -1,4 +1,4 @@
-.TH RDIFF-BACKUP 1 "JULY 2007" "Version 1.1.13" "User Manuals" \" -*- nroff -*-
+.TH RDIFF-BACKUP 1 "OCTOBER 2008" "Version 1.2.2" "User Manuals" \" -*- nroff
-*-
.SH NAME
rdiff-backup \- local/remote mirror and incremental backup
.SH SYNOPSIS
@@ -344,7 +344,7 @@
rdiff-backup-data directory. rdiff-backup will run slightly quicker
and take up a bit less space.
.TP
-.BI \-\-no-hard-links
+.B \-\-no-hard-links
Don't replicate hard links on destination side. If many hard-linked
files are present, this option can drastically decrease memory usage.
This option is enabled by default if the backup source or restore
@@ -369,6 +369,15 @@
.B \-\-override-chars-to-quote
If the filesystem to which we are backing up is not case-sensitive, automatic
'quoting' of characters occurs. For example, a file 'Developer.doc' will be
converted into ';068eveloper.doc'. To override this behavior, you need to
specify this option.
.TP
+.BI "\-\-override-quote-chars-and-fsabilities-from-file " filename
+This option lets you override ANY filesystem attributes and characters to
quote, read from a config file.
+Similar to above \-\-override-chars-to-quote, but offers advanced
functionality, you can specify INCLUDES, EXCLUDES,
+each also as a range of values (e.g. \\0x00-\\0x30) and supports
decimal/hexadecimal and octal values.
+Use this option to create a single cross-platform-compatibility file, when you
use several OS with different filesystems,
+then all filenames are converted properly to be
+.B completely cross-platform
+compatible. See below for a file-format description and example.
+.TP
.B \-\-preserve-numerical-ids
If set, rdiff-backup will preserve uids/gids instead of trying to
preserve unames and gnames. See the
@@ -485,6 +494,28 @@
in the following host::filename argument(s). The filename section
will be ignored.
.TP
+.B \-\-use-compatible-timestamps
+When \-\-use-compatible-timestamps is enabled, the above timestamp
+is created as following: "2008-10-13T04-09-38-07-00" (instead
+of default: "2008-10-13T04:09:38-07:00"). This format is allowed
+also on Windows filesystems where colons (":") are disallowed in
+filenames! If you want to locally backup a Unix path and later use
+.BR rsync (1)
+to backup to Microsoft Windows system, use this option!
+This option is fail-safe, you can continue your 'traditional' timestamped
backups, change to this option
+later and even revert to the original, non-windows-compatible timestamp or
even intermix different timestamp formats.
+.TP
+.B \-\-use-utc
+When \-\-use-utc is enabled, the timestamp is always stored as UTC,
+indicated by appended "Z" (for Zulu) instead of timezone
+offset, thus becomes "2001-07-15T04-09-38Z". Using \-\-use-utc
+is recommended when 1) a remote backup is performed, 2) source
+and destination for backup/restore are in different timezones, 3)
+you want to keep yourself from 'daylight saving' errors, 4) all
+timestamped filenames are 5 characters shorter.
+This option is fail-safe, you can continue your 'traditional' timestamped
backups, change to this option
+later and even revert to the original, non-windows-compatible timestamp or
even intermix different timestamp formats.
+.TP
.BI "\-\-user-mapping-file " filename
Map user names and ids according to the user mapping file
.IR filename .
@@ -577,6 +608,28 @@
http://www.w3.org/TR/NOTE-datetime. Basically they look like
"2001-07-15T04:09:38-07:00", which means what it looks like. The
"-07:00" section means the time zone is 7 hours behind UTC.
+When
+.B \-\-use-compatible-timestamps
+is enabled, the above timestamp is created as following:
"2001-07-15T04-09-38-07-00". This format
+is allowed also on Windows filesystems where ":" colons are disallowed in
filenames!
+This option is fail-safe, you can continue your 'traditional' timestamped
backups, change to this option
+later and even revert to the original, non-windows-compatible timestamp or
even intermix any.
+When additionally
+.B \-\-use-utc
+is enabled, then the timestamp is
+always used as UTC, indicated by appended "Z" (for Zulu) instead
+of timezone offset, thus becomes "2001-07-15T04-09-38Z". See above option
details for benefits.
+This option is fail-safe, you can continue your 'traditional' timestamped
backups, change to this option
+later and even revert to the original, non-windows-compatible timestamp or
even intermix them.
+
+Using
+.B \-\-use-utc
+is recommended when 1) a remote backup is performed, 2) source
+and destination for backup/restore are in different timezones, 3)
+you want to keep yourself from 'daylight saving' problems, 4) all
+timestamped filenames are 5 characters shorter.
+This option is fail-safe, you can continue your 'traditional' timestamped
backups including timezone-info,
+then use this option at any time, even intermixed.
.PP
Secondly, the
.BI \-r , " \-\-restore-as-of" ", and " \-\-remove-older-than
@@ -697,6 +750,7 @@
.BR \-\-include-globbing-filelist ,
.BR \-\-include-globbing-filelist-stdin ,
.BR \-\-include-filelist-stdin ,
+.BR \-\-override-quote-chars-and-fsabilities-from-file,
and
.BR \-\-include-regexp .
Each file selection condition either matches or doesn't match a given
@@ -920,6 +974,53 @@
the same as specifying "\-\-include dir/foo \-\-include dir/bar \-\-exclude **"
on the command line.
+.B "\-\---override-quote-chars-and-fsabilities-from-file override_cfg.txt"
+.RE This option lets you override ANY filesystem attributes and characters to
quote, read from a config file.
+Similar to above \-\-override-chars-to-quote, but offers advanced
functionality, you can specify INCLUDES, EXCLUDES,
+each also as a range of values (\\0x00-\\0x30) and supports
decimal/hexadecimal and octal values.
+Use this option to create a single cross-platform-compatibility file, when you
use several OS with different filesystems,
+then all filenames are converted properly to be
+.B completely cross-platform
+compatible.
+.RE
+.B File-Format:
+.RE ########################################################
+ # Format description: escaped ('\\'-prefixed) numbers,
+ # supported are octal/hexadecimal/decimal values (even
+ # inter-mixed), separated by semicolon ';'
+ # or specifying a character-range with a hyphen ('-')
+ # (e.g. \\0x00-\\0x1F). For help in conversion from characters
+ # to decimal/hexadecimal/octal values, see URLs:
+ # http://www.asciitable.com/
+ # http://www.asciitable.de/tabelle.html
+ ###########################################################
+ INCLUDE:\\0x00-\\0x1F;\\0x22;\\0x2A;\\0x2F;\\0x3A;\\0x3C;\\0x3E;\
+\\0x3F;\\0x5C;\\0x7C;\\0x7F;\\0x3B
+ # Description of above INCLUDE: Quote non-printable char-range
+ # decimal 0-31, and also quote: ", *, /, :, <, >, ?, \, |,
+ # and 127 (decimal; hex: 7F) (DEL) and ';' quotation
+ # char itself. This is required for Windows-compatibility.
+ EXCLUDE:
+.RE
+ ############################################################
+ # FS-ABILITIES: nameformat as in Python source-code, with
+ # '+' and '-' prefix to indicate enabling, resp. disabling a
+ # feature, separated by ';' from following other options.
+ # If you do not want an auto-detected filesystem-ability to
+ # be overwritten, simply remove the option from below list.
+ #
+ ## TODO: interaction and precedence from commandline with:
+ # --exclude-special-files (device-files;fifos;sockets;symbolic-links);
+ # --exclude-other-filesystems
+.RE
+ ###########
+ FS-ABILITIES:+case_sensitive;+eas;+acls;+win_acls; \
+ +escape_dos_devices;-resource_forks;-carbonfile;-hardlinks; \
+ -fsync_dirs;-change_ownership;-high_perms;-symlink_perms;
+ ############################################################
+.RE
+
+.BR
Finally, the
.B \-\-include-regexp
and
[/code]
[code]
diff U3 eas_acls.py eas_acls.py
--- eas_acls.py Sat Sep 27 01:08:31 2008
+++ eas_acls.py Sat Nov 01 11:15:45 2008
@@ -224,7 +224,7 @@
"""Set self.entry_list and self.default_entry_list from text"""
self.entry_list, self.default_entry_list = [], []
for line in text.split('\n'):
- comment_pos = text.find('#')
+ comment_pos = line.find('#') ## AFAICT was BUG before:
comment_pos = text.find('#')
if comment_pos >= 0: line = line[:comment_pos]
line = line.strip()
if not line: continue
diff U3 FilenameMapping.py FilenameMapping.py
--- FilenameMapping.py Sat Jan 05 19:43:13 2008
+++ FilenameMapping.py Thu Oct 23 14:04:21 2008
@@ -65,6 +65,7 @@
def init_quoting_regexps():
"""Compile quoting regular expressions"""
global chars_to_quote_regexp, unquoting_regexp
+
assert chars_to_quote and type(chars_to_quote) is types.StringType, \
"Chars to quote: '%s'" % (chars_to_quote,)
try:
diff U3 fs_abilities.py fs_abilities.py
--- fs_abilities.py Wed Oct 08 00:45:42 2008
+++ fs_abilities.py Tue Nov 04 13:11:27 2008
@@ -27,7 +27,7 @@
"""
-import errno, os
+import errno, os, re, string
import Globals, log, TempFile, selection, robust, SetConnections, \
static, FilenameMapping, win_acls
@@ -73,7 +73,7 @@
else:
assert boolean == 0
val_text = 'Off'
- addline(desc, val_text)
+ addline(desc, val_text)
def get_title_line():
"""Add the first line, mostly for decoration"""
@@ -164,6 +164,9 @@
subdir.delete()
return self
+
+# def show_active_fsabilities(self, ):
+
def set_ownership(self, testdir):
"""Set self.ownership to true iff testdir's ownership can be
changed"""
@@ -556,6 +559,11 @@
% (subdir.path), 4)
self.escape_dos_devices = 1
+
+## OM: Todo: Move function 'update_fs_abilities' herein?
+# def update_fs_abilities(self, key, value):
+###### END OF update_fs_abilities()
+
def get_readonly_fsa(desc_string, rp):
"""Return an fsa with given description_string
@@ -633,6 +641,9 @@
class BackupSetGlobals(SetGlobals):
"""Functions for setting fsa related globals for backup session"""
+
+#----------------------------------------------------------------------
+
def update_triple(self, src_support, dest_support, attr_triple):
"""Many of the settings have a common form we can handle here"""
active_attr, write_attr, conn_attr = attr_triple
@@ -645,6 +656,8 @@
SetConnections.UpdateGlobal(write_attr, 1)
self.out_conn.Globals.set_local(conn_attr, 1)
+#----------------------------------------------------------------------
+
def set_must_escape_dos_devices(self, rbdir):
"""If local edd or src edd, then must escape """
try:
@@ -657,21 +670,29 @@
log.Log("Backup: must_escape_dos_devices = %d" % \
(self.src_fsa.escape_dos_devices or local_edd),
4)
+#----------------------------------------------------------------------
+
def set_chars_to_quote(self, rbdir, force):
- """Set chars_to_quote setting for backup session
+ """Set chars_to_quote setting for backup session. Unlike the
other options,
+ the chars_to_quote setting also depends on the current settings
in the
+ rdiff-backup-data directory, not just the current fs features.
+ """
- Unlike the other options, the chars_to_quote setting also
- depends on the current settings in the rdiff-backup-data
- directory, not just the current fs features.
+ ctq = []
- """
- (ctq, update) = self.compare_ctq_file(rbdir,
- self.get_ctq_from_fsas(), force)
+ if Globals.is_not_None('override_chars_to_quote'):
+ wanted_ctq = self.get_ctq_and_fsabilities_from_file(
Globals.get('path_octqff') )
+ else: wanted_ctq = self.get_ctq_from_fsas()
+
+# WAS: (ctq, update) = self.compare_ctq_file(rbdir,
self.get_ctq_from_fsas(), force)
+ (ctq, update) = self.compare_ctq_file(rbdir, wanted_ctq, force)
SetConnections.UpdateGlobal('chars_to_quote', ctq)
if Globals.chars_to_quote: FilenameMapping.set_init_quote_vals()
return update
+#----------------------------------------------------------------------
+
def get_ctq_from_fsas(self):
"""Determine chars_to_quote just from filesystems, no ctq
file"""
ctq = []
@@ -684,12 +705,276 @@
if self.dest_fsa.win_reserved_filenames:
if self.dest_fsa.extended_filenames:
ctq.append('\000-\037') # Quote 0 - 31
- # Quote ", *, /, :, <, >, ?, \, |, and 127 (DEL)
+ # Quote ", *, /, :, <, >, ?, \, |, and 127 (decimal)
(DEL)
ctq.append('\"*/:<>?\\\\|\177')
- if ctq: ctq.append(';') # Quote quoting char if quoting anything
+# OM: WAS pot. bug before, because unconditionally only ';' was quoted, even
if overridden with another character
+# if ctq: ctq.append(';') # Quote quoting char if quoting anything
+ if ctq: ctq.append(Globals.get('quoting_char')) # Quote defined
quoting-char if quoting anything
return "".join(ctq)
+#########################################################################################
+## OM-test
+## TODO: Should be a combination of filesystem-detected requirements ?!?
+## Safe-version: use both detected by FS-capabilities AND from file => most
sensible
+##
+## TODO: enhance also overriding filesystem-detected capabilities; e.g.
+## use case-sensitivity even on case-insensitive FS, no hardlinking even if
supported, etc.
+## impl. here also! +Option, -Option
+##
+## TODO: later impl. option, if filesystem (SRC, DEST or intermediary!) is
case-insensitive,
+## do simple 'sliding window' test for each directory to find any potential
collision AND escape
+## only these. While writing this, a '--dry-run' option performing this action
would be great,
+## add value to this software and positively advocates on file-naming!
+## TODO: find any existing project/code to do this! There are only some regexp
required for this!
+
+
+ def get_ctq_and_fsabilities_from_file(self, filepath_ctq):
+ """Determine chars_to_quote just from file, overriding all other
settings"""
+
+ ctq, ctq_includes, ctq_excludes, custom_fs_abilities = [], [],
[], []
+
+ try:
+ file_in = open(filepath_ctq, 'r')
+ except IOError:
+ log.Log("Error: Could not open
'override-quoted-chars-and-fsabilities'-file '%s'!" % (filepath_ctq), 1)
+ else:
+ for line in file_in.readlines():
+ line = line.strip()
+ if not line: continue ## skip empty lines
+# log.Log("DEBUG: get_ctq_from_file(): line read:
'%s'\n" % line, 6)
+ if re.match("^#", line): ## skip the format
description/examples + commented out lines
+# log.Log("DEBUG: get_ctq_from_file(): comment
skipped\n", 6)
+ continue
+ elif re.match("^FS-ABILITIES:(.*?)$", line):
+ m = re.match("^FS-ABILITIES:(.*?)$",
line) ## How to include the assignment 'm = ' right in above line?
+ custom_fs_abilities = m.group(1).strip()
+ log.Log("DEBUG: get_ctq_from_file():
FS-ABILITIES: Filesystem overrides: '%s'\n" % ( custom_fs_abilities ), 6)
+
self.override_fs_abilities(custom_fs_abilities)
+ else:
+ m = re.match("^(IN|EX)CLUDE:(.*?)$",
line)
+ if m.group(1) == 'IN': ctq_includes =
self.decode_characters(m.group(2).strip())
+ elif m.group(1) == 'EX': ctq_excludes =
self.decode_characters(m.group(2).strip())
+ log.Log("DEBUG: get_ctq_from_file(): %sCLUDE
contents: '%s'\n" % (m.group(1), m.group(2)), 6)
+ file_in.close()
+
+ ctq_decimals = self.diff_ctqff(ctq_includes, ctq_excludes)
+ result = self.collapse_ctqff(ctq_decimals)
+
+# log.Log("Number of includes: '%d', of excludes '%d'" % (
len(ctq_includes), len(ctq_excludes) ), 6)
+# print "DEBUG: final DIFF-array of decimal character values (range
only 0-255): ", ctq_decimals
+ log.Log("DEBUG: Final list of chars to quote: '%s'\n" % result,
6)
+
+ return result
+
+#----------------------------------------------------------------------
+
+ def decode_characters(self, encoded_string):
+ """
+ Reads characters in multiple formats
(decimal/hexadecimal/octal), expands
+ any ranges provided and converts/unifies all to decimals for
later easy array-processing.
+ Attention: only allows ASCII (0-255), no Unicode-points at this
time!
+
+ For help in conversion from/to decimal/hexadecimal/octal
values, see URLs:
+ http://www.asciitable.com/
+ http://www.asciitable.de/tabelle.html
+ """
+
+ final_chars_to_quote_list = []
+
+ for item in encoded_string.split(';'):
+ if not item: continue
+# print "DEBUG: part '%s'" % ( item )
+
+ sublist = item.split('-')
+
+ if (len(sublist) != 2): ## is a number-representation
of a single character only
+
+ replaced_number = string.replace( sublist[0],
'\\', '')
+ ## Get the number from whatever Base-/Radix was
specified, either decimal/hexadecimal or octal
+ char_number = int(replaced_number, 0) ## read +
convert to decimal/base10
+ final_chars_to_quote_list.append( char_number )
+## For debugging:
+# if (char_number <= 255): char = chr(char_number)
+# else: char = unichr(char_number)
+# print "DEBUG: Single char appended: decimal '%d' as
char '%s'" % ( char_number, char )
+
+ else: ## a range of characters specified, represented
as numbers
+
+ replaced_number0 = string.replace( sublist[0],
'\\', '')
+ replaced_number1 = string.replace( sublist[1],
'\\', '')
+ char_number0 = int(replaced_number0, 0)
+ char_number1 = int(replaced_number1, 0)
+
final_chars_to_quote_list.extend(range(char_number0, char_number1 + 1)) ## '+1'
because range() excludes second parameter
+
+## For debugging:
+# if (char_number0 <= 255): char0 =
chr(char_number0)
+# else: char0 = unichr(char_number0)
+# if (char_number1 <= 255): char1 =
chr(char_number1)
+# else: char1 = unichr(char_number1)
+# print "DEBUG: Char range appended: decimal '%d' - '%d'
=> '%s' - '%s'" % (char_number0, char_number1, char0, char1)
+
+# print "DEBUG: char-array of decimal values: ",
final_chars_to_quote_list
+ return final_chars_to_quote_list
+
+#----------------------------------------------------------------------
+
+ def diff_ctqff(self, list_includes, list_excludes):
+ """ Actually find all decimal character values to quote, with any excluded
values removed."""
+ list_includes.sort()
+ list_excludes.sort()
+ final_list_ctqff = [item for item in list_includes if item not
in list_excludes]
+ return final_list_ctqff
+
+#----------------------------------------------------------------------
+
+ def collapse_ctqff(self, listing):
+ """
+ Collapses/compresses **more than two** consecutive integer numbers
into a range: e.g. 0 1 2 3 4 8 10 11 => 0-4,8,10,11
+ Problem: This is important or else a bug is thrown when quoted
chars in range 0-31 decimal are given (which is required for Windows FSs)
+ of type: "TypeError: 'NoneType' object is not iterable"
+ Solution: The char-range 0-31 decimal must be written to
'quote'-file with '-' as a range, as in default quote-file for Windows FSs.
+ """
+
+ final_string, str_list, quoting_char_included = '', [], 0
+ quoting_integer_number = ord(Globals.get('quoting_char')) ## If
the quoting char itself is forgotten from the list, add it!
+
+## OM: Remark: this works for now, but 'feels' not ideal ... please improve.
+ accumulator, start, stop, iter = -1, -1, -1, -1
+
+ for item in listing:
+ iter = iter + 1
+# print "DEBUG: iterating item '%s' - number '%d'" %
(item, iter)
+
+ if ((quoting_char_included == 0) and (item ==
quoting_integer_number)):
+ quoting_char_included = 1
+
+ if iter == 0: start = item; accumulator = 0; continue
+ elif (listing[iter] == (listing[iter - 1] + 1)): ##
consecutive integers, store
+# print "DEBUG: consecutive numbers found
'%r'-'%r'\n" % (listing[iter]-1, listing[iter])
+ stop = item; accumulator = accumulator + 1
+ elif accumulator >= 2: ## more than two consecutive
integers, store
+# print "DEBUG: accumulator '%s'\n" % (chr(start)
+ '-' + chr(stop))
+ str_list.append(chr(start) + '-' + chr(stop));
+ accumulator = 0; start = item
+ elif accumulator >= 1: ## only two consecutive
integers, do not store as range, but simple append both
+# print "DEBUG: accumulator '%s'\n" % (chr(start)
+ chr(stop))
+ str_list.append(chr(start) + chr(stop));
+ accumulator = 0; start = item
+ else:
+ accumulator = 0; appendix = chr(start)
+ str_list.append(appendix); start = item
+# print "DEBUG: ELSE-clause, solitaire char only '%s'
appended\n" % appendix
+ ## END for-loop
+ str_list.append(chr(start)) ## Last character needs to be
appended, too!
+
+ # Add defined quoting-char to existing list if not already
included
+ if (str_list and (quoting_char_included == 0)):
str_list.append(Globals.get('quoting_char'))
+
+ return "".join(str_list)
+
+#----------------------------------------------------------------------
+
+ def override_fs_abilities(self, string_fs_abilities):
+ """ Attention: Only use when you know what you are doing, since this might
break backups. """
+
+ """
+#OM: FS-ABILITIES: nameformat as in Python-code
+#FS-ABILITIES:+case_sensitivity;+eas;+acls;+win_acls;+escape_dos_devices;-resource_forks;-carbonfile;-hardlinks;-fsync_dirs;-change_ownership;-high_perms;-symlink_perms;
+## How about: --exclude-special-files (--exclude-device-files;
--exclude-fifos;--exclude-sockets;--exclude-symbolic-links);
--exclude-other-filesystems
+ """
+
+ fs_abilities_enabled, fs_abilities_disabled = [], []
+
+ for item in string_fs_abilities.split(';'):
+ if not item: continue
+# print "DEBUG: item '%s'" % ( item )
+ m = re.match("^(\+|-)(.*?)$", item)
+ if m.group(1) == '+':
fs_abilities_enabled.append(m.group(2))
+ elif m.group(1) == '-':
fs_abilities_disabled.append(m.group(2))
+ else: log.Log("Error! There's a '+' or '-' prefix missing
from item '%s' in fs_abilities override file, fix this first!" % (m.group(2)), 6)
+ ## END FOR-LOOP
+
+# print "DEBUG: fs_abilities ENABLED: ", fs_abilities_enabled, " -
DISABLED: ", fs_abilities_disabled
+
+ for item in fs_abilities_enabled:
self.update_fs_abilities(item, 1)
+ for item in fs_abilities_disabled:
self.update_fs_abilities(item, 0)
+
+ """
+ TODO: interaction with the corresponding commandline options?
+Globals.set("never_drop_acls", 1) ; Globals.set("carbonfile_active", 0) ;
Globals.set("acls_active", 0)
+Globals.set("win_acls_active", 0) ; Globals.set('preserve_hardlinks', 0) ;
Globals.set('resource_forks_active', 0)
+ """
+
+#----------------------------------------------------------------------
+
+ def info_updated_fsa(self, key, value):
+ print "DEBUG: update_fs_abilities(src,dest): '%s' was changed to
'%s'." % (key, value)
+ Globals.set('override_fsabilities', 1)
+
+#----------------------------------------------------------------------
+
+ def update_fs_abilities(self, key, value):
+
+# print "DEBUG: update_fs_abilities(): processing '%s'" % key
+
+## OM: AIM: Wanted to create/lookup variable from string/key:
+# I tried below to no avail with and without 'self.' prefix:
+# src_key = 'self.src_fsa.' + key
+# src_key = vars()['self.src_fsa.' + key]
+# src_key = locals()['self.src_fsa.' + key]
+# src_key = globals()['self.src_fsa.' + key]
+# eval(src_key = 'self.src_fsa.' + key)
+# exec( "src_key = 'self.src_fsa.' + key")
+#
+## OM: This is an UGLY KLUDGE/workaround for now. AIM: Wanted to create/lookup
variable from string/key:
+# src_key = "self.src_fsa." + key; print "Src-key: '%s'\n" %
src_key
+
+ fsa_attribute_changed = 1
+
+ if ( (key.find( 'extended_filenames' ) != -1) and (
self.src_fsa.extended_filenames != value ) ):
+ self.src_fsa.extended_filenames =
self.dest_fsa.extended_filenames = value
+ elif ( (key.find( 'win_reserved_filenames' ) != -1) and (
self.src_fsa.win_reserved_filenames != value) ):
+ self.src_fsa.win_reserved_filenames =
self.dest_fsa.win_reserved_filenames = value
+ elif ( (key.find( 'case_sensitive' ) != -1) and
(self.src_fsa.case_sensitive != value) ):
+ self.src_fsa.case_sensitive =
self.dest_fsa.case_sensitive = value
+ elif ( (key.find( 'ownership' ) != -1) and
(self.src_fsa.ownership != value) ):
+ self.src_fsa.ownership = self.dest_fsa.ownership = value
+ elif ( (key.find( 'eas' ) != -1) and (self.src_fsa.eas !=
value) ):
+ self.src_fsa.eas = self.dest_fsa.eas = value
+ elif ( (key.find( 'win_acls' ) != -1) and
(self.src_fsa.win_acls != value) ):
+ self.src_fsa.win_acls = self.dest_fsa.win_acls = value
+ elif ( (key.find( 'acls' ) != -1) and (self.src_fsa.acls !=
value) ): ### IMPORTANT! 'acls' check must be after 'win_acls' check!
+ self.src_fsa.acls = self.dest_fsa.acls = value
+ elif ( (key.find( 'hardlinks' ) != -1) and
(self.src_fsa.hardlinks != value) ):
+ self.src_fsa.hardlinks = self.dest_fsa.hardlinks = value
+ elif ( (key.find( 'fsync_dirs' ) != -1) and
(self.src_fsa.fsync_dirs != value) ):
+ self.src_fsa.fsync_dirs = value ## Not needed:
self.dest_fsa.fsync_dirs
+ elif ( (key.find( 'dir_inc_perms' ) != -1) and
(self.src_fsa.dir_inc_perms != value) ):
+ self.src_fsa.dir_inc_perms =
self.dest_fsa.dir_inc_perms = value
+ elif ( (key.find( 'resource_forks' ) != -1) and
(self.src_fsa.resource_forks != value) ):
+ self.src_fsa.resource_forks =
self.dest_fsa.resource_forks = value
+ elif ( (key.find( 'carbonfile' ) != -1) and
(self.src_fsa.carbonfile != value) ):
+ self.src_fsa.carbonfile = self.dest_fsa.carbonfile =
value
+ elif ( (key.find( 'high_perms' ) != -1) and
(self.src_fsa.high_perms != value) ):
+ self.src_fsa.high_perms = self.dest_fsa.high_perms =
value
+ elif ( (key.find( 'escape_dos_devices' ) != -1) and
(self.src_fsa.escape_dos_devices != value) ):
+ self.src_fsa.escape_dos_devices_dirs =
self.dest_fsa.escape_dos_devices = value
+ elif ( (key.find( 'symlink_perms' ) != -1) and
(self.src_fsa.symlink_perms != value) ):
+ self.src_fsa.symlink_perms =
self.dest_fsa.symlink_perms = value
+ else:
+ fsa_attribute_changed = 0
+ print "DEBUG: update_fs_abilities(): Option '%s' either not
changed or not recognized - skipped!" % key
+
+ if (fsa_attribute_changed != 0): self.info_updated_fsa(key,
value)
+
+## OM: END of UGLY KLUDGE/workaround.
+
+#----------------------------------------------------------------------
+
+#########################################################################################
+## END OM-test
+
def compare_ctq_file(self, rbdir, suggested_ctq, force):
"""Compare ctq file with suggested result, return actual ctq"""
ctq_rp = rbdir.append("chars_to_quote")
@@ -733,9 +1018,11 @@
repository from the old quoting chars to the new ones.""" %
(suggested_ctq, actual_ctq, ctq_rp.path))
+#----------------------------------------------------------------------
class RestoreSetGlobals(SetGlobals):
"""Functions for setting fsa-related globals for restore session"""
+
def update_triple(self, src_support, dest_support, attr_triple):
"""Update global settings for feature based on fsa results
@@ -754,6 +1041,8 @@
self.out_conn.Globals.set_local(write_attr, 1)
if src_support: self.in_conn.Globals.set_local(conn_attr, 1)
+#----------------------------------------------------------------------
+
def set_must_escape_dos_devices(self, rbdir):
"""If local edd or src edd, then must escape """
if getattr(self, "src_fsa", None) is not None:
@@ -769,6 +1058,8 @@
log.Log("Restore: must_escape_dos_devices = %d" % \
(src_edd or local_edd), 4)
+#----------------------------------------------------------------------
+
def set_chars_to_quote(self, rbdir):
"""Set chars_to_quote from rdiff-backup-data dir"""
if Globals.chars_to_quote is not None: return # already
overridden
@@ -781,9 +1072,11 @@
"assuming no quoting in backup
repository.", 2)
SetConnections.UpdateGlobal("chars_to_quote", "")
+#----------------------------------------------------------------------
class SingleSetGlobals(RestoreSetGlobals):
"""For setting globals when dealing only with one filesystem"""
+
def __init__(self, conn, fsa):
self.conn = conn
self.dest_fsa = fsa
@@ -816,6 +1109,7 @@
self.update_triple(self.dest_fsa.carbonfile,
('carbonfile_active', 'carbonfile_write',
'carbonfile_conn'))
+#----------------------------------------------------------------------
def backup_set_globals(rpin, force):
"""Given rps for source filesystem and repository, set fsa globals
@@ -848,6 +1142,10 @@
if update_quoting and force:
FilenameMapping.update_quoting(Globals.rbdir)
+ if Globals.is_not_None('override_fsabilities'):
+ print "Changed Filesystem-Attributes:\nSRC-FSA:\n", src_fsa,
"\nDEST-FSA:\n", dest_fsa
+
+#----------------------------------------------------------------------
def restore_set_globals(rpout):
"""Set fsa related globals for restore session, given in/out rps"""
@@ -873,6 +1171,8 @@
rsg.set_escape_dos_devices()
rsg.set_must_escape_dos_devices(Globals.rbdir)
+#----------------------------------------------------------------------
+
def single_set_globals(rp, read_only = None):
"""Set fsa related globals for operation on single filesystem"""
if read_only:
@@ -894,3 +1194,4 @@
ssg.set_escape_dos_devices()
ssg.set_must_escape_dos_devices(Globals.rbdir)
+#----------------------------------------------------------------------
diff U3 Globals.py Globals.py
--- Globals.py Wed Jul 02 19:03:23 2008
+++ Globals.py Tue Nov 04 12:51:56 2008
@@ -23,12 +23,15 @@
# The current version of rdiff-backup
-version = "$version"
+version = "$version"
# If this is set, use this value in seconds as the current time
# instead of reading it from the clock.
current_time = None
+# Stores the local timezone for later UTC-conversion
+local_timezone = None
+
# This determines how many bytes to read at a time when copying
blocksize = 131072
@@ -158,6 +161,27 @@
# should be escaped (see FilenameMapping for more info).
chars_to_quote = None
quoting_char = ';'
+
+##############################################################################
+## OM-test
+## If set, the timestamps use the following format: "2008-09-01T04-49-04-07-00"
+## (instead of "2008-09-01T04:49:04-07:00"). This creates truly cross-platform
+## timestamps, e.g. required for transferring rdiff-backup archive to Windows
filesystems.
+use_compatible_timestamps = None
+
+## if set, uses the characters/ranges for overriding local + evg. remote
filesystem attributes
+## detected. Useful when you need to transfer a locally done backup later to
another (e.g. Windows) filesystem.
+## Specify in this file either the characters you want to replace or their
octal values + ranges (e.g. '\000-\030').
+override_chars_to_quote = None
+path_octqff = None
+override_fsabilities = None
+## TODO: merge with above chars_to_quote var?!?
+
+## If set, use UTC as base for all file-access/modification calculations.
Removes the explicit
+## timezone-offset, and appends 'Z' for Zulu to the timestamp, resulting in
"2008-09-01T04-49-04-Z"
+use_utc = None
+## END OM-test
+##############################################################################
# If true, emit output intended to be easily readable by a
# computer. False means output is intended for humans.
diff U3 Main.py Main.py
--- Main.py Sun Oct 12 16:46:00 2008
+++ Main.py Sun Nov 02 14:34:31 2008
@@ -85,8 +85,11 @@
"remove-older-than=", "restore-as-of=", "restrict=",
"restrict-read-only=", "restrict-update-only=", "server",
"ssh-no-compression", "tempdir=", "terminal-verbosity=",
- "test-server", "user-mapping-file=", "verbosity=", "verify",
- "verify-at-time=", "version"])
+ "test-server",
+ ## OM-test
+ "use-utc", "use-compatible-timestamps",
"override-quote-chars-and-fsabilities-from-file=",
+ ## END
+ "user-mapping-file=", "verbosity=", "verify", "verify-at-time=",
"version"])
except getopt.error, e:
commandline_error("Bad commandline options: " + str(e))
@@ -146,6 +149,25 @@
select_opts.append(("--include-globbing-filelist",
"standard
input"))
select_files.append(sys.stdin)
+
+ ## OM-test
+ ## Advantages: --use-utc: local/remote systems in different
timezones do not cause problems
+ elif opt == "--override-quote-chars-and-fsabilities-from-file":
+ p_octqff = arg
+ Globals.set_integer('override_chars_to_quote', 1)
+ Globals.set('path_octqff', p_octqff)
+# print "DEBUG: override_chars_to_quote-from-file '%s'" %
p_octqff
+
+# BackupSetGlobals.get_ctq_from_file(
Globals.get('path_octqff') )
+# ## was: fs_abilities.get_ctq_from_file()
+# REM: required also? Globals.set('chars_to_quote', arg)
+
+ elif opt == "--use-utc":
+ Globals.set("use_utc", 1)
+ elif opt == "--use-compatible-timestamps":
+ Globals.set("use_compatible_timestamps", 1)
+ ## END OM-test
+
elif opt == "--include-regexp": select_opts.append((opt, arg))
elif opt == "--list-at-time":
restore_timestr, action = arg, "list-at-time"
@@ -240,8 +262,7 @@
else: action = "backup"
def commandline_error(message):
- Log.FatalError(message + "\nSee the rdiff-backup manual page for "
- "more information.")
+ Log.FatalError(message + "\nSee the rdiff-backup manual page for more
information.")
def misc_setup(rps):
"""Set default change ownership flag, umask, relay regexps"""
@@ -476,6 +497,9 @@
Log.open_logfile(Globals.rbdir.append("backup.log"))
checkdest_if_necessary(rpout)
prevtime = backup_get_mirrortime()
+
+ Log("DEBUG: prevtime is %s" % prevtime, 6)
+
if prevtime >= Time.curtime: Log.FatalError(
"""Time of Last backup is not in the past. This is probably caused
by running two backups in less than a second. Wait a second a try again.""")
diff U3 metadata.py metadata.py
--- metadata.py Sat Sep 27 01:17:24 2008
+++ metadata.py Tue Oct 21 10:08:28 2008
@@ -542,10 +542,13 @@
def _writer_helper(self, prefix, flatfileclass, typestr, time):
"""Used in the get_xx_writer functions, returns a writer
class"""
+
if time is None: timestr = Time.curtimestr
else: timestr = Time.timetostring(time)
filename = '%s.%s.%s' % (prefix, timestr, typestr)
rp = Globals.rbdir.append(filename)
+
+ log.Log("DEBUG: created filename '%s'\n" % filename, 6)
assert not rp.lstat(), "File %s already exists!" % (rp.path,)
assert rp.isincfile()
return flatfileclass(rp, 'w', callback = self.add_incrp)
diff U3 rpath.py rpath.py
--- rpath.py Sun Oct 12 16:46:00 2008
+++ rpath.py Tue Oct 21 14:09:51 2008
@@ -352,10 +352,14 @@
assert rpath.conn is Globals.local_connection
return open(rpath.path, "rb")
+## This fn is erroneous? No! This returns the number of tokens!!
def get_incfile_info(basename):
"""Returns None or tuple of
(is_compressed, timestr, type, and basename)"""
dotsplit = basename.split(".")
+
+# log.Log("DEBUG: get_incfile_info(%s): dotsplit: '%d'" % (basename,
len(dotsplit) ), 6)
+
if dotsplit[-1] == "gz":
compressed = 1
if len(dotsplit) < 4: return None
@@ -364,7 +368,13 @@
compressed = None
if len(dotsplit) < 3: return None
timestring, ext = dotsplit[-2:]
+
+ result_time = Time.stringtotime(timestring)
+# log.Log("DEBUG: timestring: '%s' - ext '%s' - resulting timestring
'%s'" % (timestring, ext, result_time ), 6)
+
if Time.stringtotime(timestring) is None: return None
+
+
if not (ext == "snapshot" or ext == "dir" or
ext == "missing" or ext == "diff" or ext == "data"):
return None
@@ -1181,12 +1191,15 @@
def isincfile(self):
"""Return true if path looks like an increment file
-
Also sets various inc information used by the *inc* functions.
-
"""
+
+# log.Log("DEBUG: inside isincfile() beginning", 9)
+
if self.index: basename = self.index[-1]
else: basename = self.base
+
+# log.Log("DEBUG: basename is '%s'" % basename, 6)
inc_info = get_incfile_info(basename)
diff U3 Time.py Time.py
--- Time.py Thu Jan 25 04:09:16 2007
+++ Time.py Tue Nov 04 12:50:55 2008
@@ -20,8 +20,9 @@
"""Provide time related exceptions and functions"""
import time, types, re, sys, calendar
+import os ## For timezone-change
import Globals
-
+import log
class TimeException(Exception): pass
@@ -60,11 +61,40 @@
global prevtime, prevtimestr
prevtime, prevtimestr = timeinseconds, timestr
+
+## OM: currently testing which version is more cross-platform compatible:
either this fn. or time.gmtime()-call
+def set_zimezone_to_utc(reset_to_localtime_again=False):
+ """Alternative way to set timezone to UTC and reset it again to local timezone for all
timestamps"""
+ if reset_to_localtime_again:
+ os.environ['TZ'] = Globals.get(local_timezone) ##
os.environ['TZ']='Europe/Berlin'
+ else:
+ Globals.set('local_timezone', time.tzname)
+ os.environ['TZ']='' ### is UTC now
+
+ time.tzset()
+ log.Log("DEBUG: timezone used now '%s' - before was: '%s' " %
(time.tzname, Globals.get(local_timezone) ), 7)
+
+
def timetostring(timeinseconds):
"""Return w3 datetime compliant listing of timeinseconds"""
- s = time.strftime("%Y-%m-%dT%H:%M:%S", time.localtime(timeinseconds))
+ """OR else use the combatible version e.g. for Windows"""
+
+ if Globals.is_not_None('use_compatible_timestamps'): time_formatstring =
"%Y-%m-%dT%H-%M-%S"
+ else: time_formatstring = "%Y-%m-%dT%H:%M:%S"
+
+ debug_utc_used = None
+ if Globals.is_not_None('use_utc'):
+ seconds = time.gmtime(timeinseconds) ## opposite direction:
seconds = calendar.timegm(timeinseconds)
+ debug_utc_used = 1
+ else:
+ seconds = time.localtime(timeinseconds)
+
+ s = time.strftime(time_formatstring, seconds)
+# log.Log("DEBUG: Time.py: time_string is '%s' (UTC: '%s')" % (s,
debug_utc_used), 8)
+
return s + gettzd(timeinseconds)
+
def stringtotime(timestring):
"""Return time in seconds from w3 timestring
@@ -72,10 +102,19 @@
like a w3 datetime string, return None.
"""
+
+## OM: AIM: The processing of separators should be fail-safe, even mixed
separators
+## should be properly recognized and handled!
+
+ regexp = re.compile( '[-:_]' ) ## WAS: "-"
+# log.Log("DEBUG: timestring to process is '%s'" % (timestring), 9 )
+
try:
date, daytime = timestring[:19].split("T")
year, month, day = map(int, date.split("-"))
- hour, minute, second = map(int, daytime.split(":"))
+
+## OM: re.split() is required, using 'string'.split() is NOT possible with
regexp-char group.
+ hour, minute, second = map(int, re.split(regexp, daytime) )
assert 1900 < year < 2100, year
assert 1 <= month <= 12
assert 1 <= day <= 31
@@ -144,11 +183,16 @@
"""Return w3's timezone identification string.
Expresed as [+/-]hh:mm. For instance, PDT is -07:00 during
- dayling savings and -08:00 otherwise. Zone is coincides with what
+ dayling savings and -08:00 otherwise. Zone coincides with what
localtime(), etc., use. If no argument given, use the current
time.
"""
+
+ if Globals.is_not_None('use_utc'):
+# log.Log("DEBUG: Globals(use_utc) is set, instantly returning 'Z'
('Zulu' UTC identifier).", 7)
+ return 'Z'
+
if timeinseconds is None: timeinseconds = time.time()
dst_in_effect = time.daylight and time.localtime(timeinseconds)[8]
if dst_in_effect: offset = -time.altzone/60
@@ -157,16 +201,21 @@
elif offset < 0: prefix = "-"
else: return "Z" # time is already in UTC
+ if Globals.is_not_None('use_compatible_timestamps'): time_separator =
'-'
+ else: time_separator = ':'
+ log.Log("DEBUG: timezone_format separator used: '%s'.\n" %
time_separator, 8)
+
hours, minutes = map(abs, divmod(offset, 60))
assert 0 <= hours <= 23
assert 0 <= minutes <= 59
- return "%s%02d:%02d" % (prefix, hours, minutes)
+ return "%s%02d%s%02d" % (prefix, hours, time_separator, minutes)
def tzdtoseconds(tzd):
"""Given w3 compliant TZD, return how far ahead UTC is"""
+ """Now accepts both ':' and '-' separators"""
if tzd == "Z": return 0
assert len(tzd) == 6 # only accept forms like +08:00 for now
- assert (tzd[0] == "-" or tzd[0] == "+") and tzd[3] == ":"
+ assert (tzd[0] == "-" or tzd[0] == "+") and ( tzd[3] == ":" or tzd[3] == "-" or
tzd[3] == "_" )
return -60 * (60 * int(tzd[:3]) + int(tzd[4:]))
def cmp(time1, time2):
@@ -239,6 +288,8 @@
match = _genstr_date_regexp1.search(timestr) or \
_genstr_date_regexp2.search(timestr)
if not match: error()
+
+ ## OM: No need to change below format from ':', because not used for
filenames and stringtotime() was made failsafe.
timestr = "%s-%02d-%02dT00:00:00%s" % (match.group('year'),
int(match.group('month')),
int(match.group('day')), gettzd())
t = stringtotime(timestr)
[/code]
Cheers,
Oliver Mulatz
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [rdiff-backup-users] PATCHES for Cross-Platform-Compatibility (finally),
sirafil <=