[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] Show volume Duplicity is working on
From: |
Remy van Elst |
Subject: |
Re: [Duplicity-talk] Show volume Duplicity is working on |
Date: |
Sat, 18 Oct 2014 18:35:32 +0200 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 |
On 10/18/14 15:26, address@hidden wrote:
> On 18.10.2014 15:13, Remy van Elst wrote:
>>
>>
>> On 10/18/14 14:49, address@hidden wrote:
>>> 1. exactly what are you trying to achieve?
>>
>> I have a server with 1.6 TB of data that takes almost a week to do a
>> full backup over an umetered gbit connection. It backs up to Rackspace
>> Openstack Swift, with a default 25MB volume size.
>>
>> The backup becomes inconsistent because stuff changes a lot on that
>> server. It has multiple +100 GB databases running. I sent an email
>> earlier to this list regarding --max-blocksize, but did not receive a
>> response yet.
>
> you are doing snapshots or dumps of those right?
For the databases? Yes, this is one of the slaves, before Duplicity runs
pg_dump does it's trick. The actual data folders are excluded from the
duplicity run, the dump is made every 4 hours.
>
>> (http://lists.nongnu.org/archive/html/duplicity-talk/2014-10/msg00002.html)
>>
>> With a volume size of 250 MB, or with 2048 MB the upload still takes
>> more than 3 days sadly.
>>
>> What I want to achieve is solving the long running backup issue.
>
> do the backup to a local file:// target. that'd should be fast, at least as
> fast as possible. then sync this with a swift client to your backend.
>
This seems to be a good idea, I'm going to try that. Thanks :)
>>> 2. why do you need to know which volume is uploaded?
>>
>> I was troubleshooting with strace and saw the volume number. I know the
>> volume size, so with the current volume number I know which one is being
>> uploaded (async upload enabled). With that data I should be able to
>> estimate how far the backup is.
>
> for that you can enable verbose logging or progress meter, although the
> latter is sometimes not working properly.
>
The progress option results in Duplicity segfaulting after +16 hours
without uploading anything or earlier when the OOM killer comes along.
This particular machine has 256G RAM.
>>> 3. duplicity is written in python script language. you can easily patch
>>> your local duplicity to print out the current volume name.
>>>
>>
>> Is also an option, however I would use that as a last resort. I'm using
>> Duplicity 0.6.24, compiled from source on Debian 7.
>
> try the solution, separate upload, above. that way the backup will not have
> to wait for the upload top finish.
>
Going to do that, thanks again :).
> ..ede/duply.net
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>
0x1B7F88DC.asc
Description: application/pgp-keys
signature.asc
Description: OpenPGP digital signature