Re: [Duplicity-talk] Moving old sets to cold storage (listing files to be deleted)

[Ladislav Laska]

> feel free to offer a patch to duplicity to (re)enable what you need ;)

It's already on my todo-list, I'll take a look during the weekend to see how
deep would I need to dig. I might post here for some pointers if I decide to do
something about it soon.

> > Well, listing uses only signatures, as you said, and I think it won't even
> > notice if volumes are missing. 
> 
> it should.. on every run duplicity builds a list of existing chains. run in 
> max. verbosity "-v9" and you'll see.

I just did a crude test: backed up a 100M directory, removed one difftar from
the backup.

It kind of noticed that the volume is gone in collection-status. Not in a way 
I'd expect though, it
just wrote:

Total number of contained volumes: 3

Instead of 4. I removed volume 2. One might think it could have noticed that
volume 2 is missing! The -v9 output does not say anything about volume 2, so I
guess it just looks for what's there, and doesn't check for what's not.

Is this a feature? I would consider it a bug.

Moreover, list-current-files happily lists all the files, even though the
volume 2 is missing and the files are not accessible. This is what I expected,
as it's reasonable to have this cached...

So to run verify/restore seems to be the best option for checking.

>  
> > To verify missing or corrupt volumes, I really want to avoid restoring all 
> > the
> > files and some obscurities:
> > 
> > Suppose I use duplicity verify with time after the last incremental backup I
> > just moved. It would extract volumes and check with other files. For 
> > example,
> > I'm not sure what happens if:
> > 
> > Volume xyz is missing, but all the files in the volume (can be easily a
> > single big file) have been deleted. There is no need to extract it, as it 
> > should
> > not exist at the specific time.
> 
> duplicity holds a checksum for every volume file (in the manifest i think), 
> so if a volume is opened, it is checked against the checksum first and a 
> error(or at least warning) is printed.
>  
> > The same applies if xyz is corrupted (incomplete after upload), the file has
> > changed completely and has newer copy somewhere in other archive...
> 
> how.. file names are unique because of the time stamps.

I don't have an answer, as I need to familiarize myself with how the files are
stored in duplicity archives:-)

> 
> > I'm really not sure what happens in this case, and haven't found an answer, 
> > but
> > unless someone tells me otherwise, I'd have to run verify/restore once for 
> > every
> > incremental backup, and I'm not going to do that.
> > 
> > It would actually be great if I could do backend check and ensure that all 
> > the
> > files that may be needed in the future are on my backend and intact.
> 
> that's not really duplicity's purview. duplicity treats backends as dumb, to 
> make integrating backends as easy as possible.

Sure, that's great, but it's also a reason why duplicity should be able to check
if the backend contains all expected data, isn't it? After all, the backend is
dumb and we can't expect it to do it for us...

> 
> disadvantage of that approach of course is that there is no backend logic as 
> such.
> 
> what is currently called verify is actually a compare against the datasource. 
> a mere verify would test exactly what you want, but unfortunately the famous 
> no one didn't contribute it so far.

Yeah, and someone really should. I'll try to take a look, but unfrotunately, I
have more pressing work to do for the next few months.


-- 
S pozdravem Ladislav "Krakonoš" Láska                http://www.krakonos.org/