[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[rdiff-backup-users] Rewriting metadata to use arbitrary storage system

From: rdiff
Subject: [rdiff-backup-users] Rewriting metadata to use arbitrary storage system (SQLite)?
Date: Tue, 26 Jun 2007 12:58:46 -0700 (PDT)

I have been giving thoughts to breaking rdiff-backup's meta-data code into a normalized abstraction to unlink rdiff-backup from its meta-data storage. This would allow meta-data to be stored in an SQL server, DBM, flat file (current) or quantum goo (future?).

Creating meta-data interfaces for different SQL implementations such as SQLite or postgresql/mysql for bigger centralized systems could offer the flexibility which rdiff-backup really needs. The ability to run queries on the meta-data might also give us the flexibility to detect directory renames. Also think here about commit/rollback for real ACID transaction compliance (SQL junkies can druel here ;). This would truly increase the solidity of rdiff-backup.

******** With this in mind, I propose that we release a "stable" release and apply bugfixes to it as appropriate. The latest release seems fairly stable according to the list. Is it time to move 1.1.x to 1.2.x and create a new 1.3.x development fork for the more drastic changes which people are beginning to need. ********

Directory rename detection would be an AWESOME feature. We have one 3TB repository and several >100GB repositories backing up over the Internet with anywhere from 1GB to 100GB of changes per night. When someone renames a 500GB folder it (1) takes time to figure out that is what happened and (2) more time to fix the problem by either (a) blowing away the increments, renaming the folder by hand and doing an initial sync or (b) driving the backup server over to the system-to-be-backed up and re-sync everything.

FYI, at 60mbit, the 3TB backup took 6 days to do an initial sync! It was a gigabit link, but the system we were pulling from wasn't fast enough to utilize the whole pipe, and we sat at ~60mbit. I am pleased to announce that it is quite stable though, now that the initial sync has completed which does say something for rdiff-backup's scalability!



reply via email to

[Prev in Thread] Current Thread [Next in Thread]