[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Duplicity-talk] Show volume Duplicity is working on

From: Remy van Elst
Subject: Re: [Duplicity-talk] Show volume Duplicity is working on
Date: Sat, 18 Oct 2014 18:35:32 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0

On 10/18/14 15:26, address@hidden wrote:
> On 18.10.2014 15:13, Remy van Elst wrote:
>> On 10/18/14 14:49, address@hidden wrote:
>>> 1. exactly what are you trying to achieve? 
>> I have a server with 1.6 TB of data that takes almost a week to do a
>> full backup over an umetered gbit connection. It backs up to Rackspace
>> Openstack Swift, with a default 25MB volume size.
>> The backup becomes inconsistent because stuff changes a lot on that
>> server. It has multiple +100 GB databases running. I sent an email
>> earlier to this list regarding --max-blocksize, but did not receive a
>> response yet.
> you are doing snapshots or dumps of those right?

For the databases? Yes, this is one of the slaves, before Duplicity runs
pg_dump does it's trick. The actual data folders are excluded from the
duplicity run, the dump is made every 4 hours.

>> (http://lists.nongnu.org/archive/html/duplicity-talk/2014-10/msg00002.html)
>> With a volume size of 250 MB, or with 2048 MB the upload still takes
>> more than 3 days sadly.
>> What I want to achieve is solving the long running backup issue.
> do the backup to a local file:// target. that'd should be fast, at least as 
> fast as possible. then sync this with a swift client to your backend.

This seems to be a good idea, I'm going to try that. Thanks :)

>>> 2. why do you need to know which volume is uploaded?
>> I was troubleshooting with strace and saw the volume number. I know the
>> volume size, so with the current volume number I know which one is being
>> uploaded (async upload enabled). With that data I should be able to
>> estimate how far the backup is.
> for that you can enable verbose logging or progress meter, although the 
> latter is sometimes not working properly.

The progress option results in Duplicity segfaulting after +16 hours
without uploading anything or earlier when the OOM killer comes along.
This particular machine has 256G RAM.

>>> 3. duplicity is written in python script language. you can easily patch 
>>> your local duplicity to print out the current volume name.
>> Is also an option, however I would use that as a last resort. I'm using
>> Duplicity 0.6.24, compiled from source on Debian 7.
> try the solution, separate upload, above. that way the backup will not have 
> to wait for the upload top finish.

Going to do that, thanks again :).

> ..ede/duply.net
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk

Attachment: 0x1B7F88DC.asc
Description: application/pgp-keys

Attachment: signature.asc
Description: OpenPGP digital signature

reply via email to

[Prev in Thread] Current Thread [Next in Thread]