[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Duplicity-talk] (Option to) Cache retrieved volumes between (fetchi

From: edgar . soldin
Subject: Re: [Duplicity-talk] (Option to) Cache retrieved volumes between (fetching/restoring) runs locally?!
Date: Wed, 10 Nov 2010 12:12:02 +0100
User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20101027 Thunderbird/3.1.6

On 10.11.2010 00:22, Daniel Hahler wrote:
> Hello,
>> It would be much easier to simply mirror you repository to a local path and 
>> restore from there.
> It would be ~30GB, which would take a while to mirror, apart from that
> you might not have the space available.

if you are lacking the space, you probably have no space for the cache either.

>> Can't you just fetch/restore all files/folders in one go?
> Does "duply fetch" / duplicity support fetching multiple files? It does
> not look so from the man page.

Indeed it doesn't. I always expected duplicity to work with exclude/include 
also on restoring. It doesn't. Currently you can obviously only restore all or 
one file/folder. For the latter. It is the folder only without it's contents ;).

You could of course restore all to a temp folder and move what you need to 
wherever needed. That's what I did whenever I had to restore.

> If you're referring to get the meat of it only, that would have been the
> root directory of all virtual containers (which is >90% of the whole
> backup).

when it is 90% of the backup, the last 10% download really do not weigh out the 
speed improvement once you restore from your local 'cache'.

Eventually I think it might be more important to make restore respect 
in/exclude, as a user would expect it to, instead of working around it by 
looping/caching over the backup.
Would you create a feature request on launchpad for that?

regards ..ede/duply.net

> Thanks,
> Daniel
>> On 09.11.2010 21:53, Daniel Hahler wrote:
>>> Hello,
>>> I would like to be able to cache retrieved files from the backend
>>> locally between multiple runs of duplicity, e.g. via some config or
>>> command line option.
>>> Use case: having accidentally overwritten a lot of my (virtual)
>>> containers files, I've used the following to restore the previous state:
>>>   for i in $=VCIDS; do
>>>     b=path/to/files
>>>     /bin/rm -rf /$b*
>>>     duply profile fetch $b /$b 1D
>>>     duply profile fetch ${b}.d /${b}.d 1D
>>>   done
>>> This makes up 60+ runs of duplicity (2 runs of duplicity per 30+
>>> containers, one for a single file, the other for a directory), and when
>>> looking at it with "--verbosity 9" it looks like a lot of the same
>>> volumes (with a size of 50M in my case) are downloaded every time.
>>> I think it would speed this (particular) use case dramatically up, if
>>> these files would get cached locally.
>>> I could imagine to configure something like "keep files for X hours",
>>> and when duplicity gets run and files are older than this, they get
>>> cleaned on shutdown.
>>> When being accessed, they would get touched to reset the timer.
>>> However, there should also be a maximum number of files to cache, since
>>> this might easily fill your local volume otherwise.
>>> I am thinking about caching the files encrypted (just as on the remote
>>> site), but maybe caching decrypted files would make sense, too?
>>> Obviously, this should take into account if this is a remote backup
>>> (maybe by looking at the transfer rate of the files?!), and not pollute
>>> the cache, if the backend is as fast as local transfers might be.
>>> What do you think?
>>> Cheers,
>>> Daniel
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk

reply via email to

[Prev in Thread] Current Thread [Next in Thread]