Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?

[Remy van Elst]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 05/29/2015 12:58 PM, address@hidden wrote:
> On 29.05.2015 12:44, Remy van Elst wrote:
>> 
>> 
>> On 05/29/2015 12:35 PM, address@hidden wrote:
>>> On 29.05.2015 04:41, Remy van Elst wrote:
>>>> 
>>>> 
>>>> On 05/29/2015 12:25 AM, address@hidden wrote:
>>>>> please do not top post if the answer was posted at the
>>>>> bottom.. i will continue below
>>>> 
>>>>> On 28.05.2015 19:47, Remy van Elst wrote:
>>>>>> Here's the first 31000 lines of the log: 
>>>>>> https://e75f5515c4f249ffa1122853af811b3e.objectstore.eu/duplicity
/d
>>
>>>>>> 
up
>>>> 
>>>>>> 
>> lic
>>>>>> 
>>>>>> 
>>>> ity.txt
>>>>>> 
>>>>>> After these, it is all lines comparing the files on the 
>>>>>> machine. That is still running. Is this helpfull?
>>>>>> 
>>>>>> On 05/28/2015 03:54 PM, address@hidden wrote:
>>>>>>> On 28.05.2015 15:28, Remy van Elst wrote:
>>>>>>>> Any suggestions or other help?
>>>>>>>> 
>>>>>>>> On 05/25/2015 11:04 AM, Remy van Elst wrote:
>>>>>>>>> Duplicity seems to loose it's status when trying to
>>>>>>>>> do a incremental backup set over 14 days. The
>>>>>>>>> backup has been running for a while now but it is
>>>>>>>>> using about 1.5 TB of backend storage since 2
>>>>>>>>> weeks. It is going to an Openstack swift object
>>>>>>>>> store as backend.
>>>>>>>> 
>>>>>>>>> This is the command line:
>>>>>>>> 
>>>>>>>>> duplicity --asynchronous-upload --verbosity 9 
>>>>>>>>> --log-file /var/log/duplicity.log --volsize 25 
>>>>>>>>> --tempdir="/tmp" 
>>>>>>>>> --file-prefix="vps1.sparklingclouds.nl." 
>>>>>>>>> --name="vps1.sparklingclouds.nl." 
>>>>>>>>> --exclude-device-files 
>>>>>>>>> --exclude-globbing-filelist=/etc/duplicity-backup/exclude.conf
>>>>>>>>>
>>>>>>>>>
>>>>
>>>>>>>>>
>>
>>>>>>>>> 
- --full-if-older-than="14D" --no-encryption  /
>>>>>>>>> swift://duplicity-backup
>>>>>>>> 
>>>>>>>>> This is what caught my eye in the verbose log
>>>>>>>>> file:
>>>>>>>> 
>>>>>>>>> File 
>>>>>>>>> vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol166
.d
>>
>>>>>>>>> 
if
>>>> 
>>>>>>>>> 
>> ft
>>>>>> 
>>>>>>>>> 
>>>> ar
>>>>>>>> 
>>>>>>>>> 
>>>>>> .g
>>>>>>>> 
>>>>>>>> 
>>>>>>>> z
>>>>>>>>> is part of known set File 
>>>>>>>>> vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol166
0.
>>
>>>>>>>>> 
di
>>>> 
>>>>>>>>> 
>> ff
>>>>>> 
>>>>>>>>> 
>>>> ta
>>>>>>>> 
>>>>>>>>> 
>>>>>> r.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> gz
>>>>>>>>> is part of known set Found backup chain [Tue Mar 24
>>>>>>>>>  14:42:44 2015]-[Tue Mar 24 14:42:44 2015] Last
>>>>>>>>> full backup is too old, forcing full backup
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> The last full backup is from yesterday, and it
>>>>>>>>> should only do a full backup every 14 days.
>>>>>>>> 
>>>>>>>>> When I do a collection status it seems all the 
>>>>>>>>> incremental backups are gone:
>>>>>>>> 
>>>>>>>>> Do note that I've got a wrapper script around 
>>>>>>>>> duplicity which outputs the status, storage used
>>>>>>>>> and config settings:
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> # Hostname: vps1.sparklingclouds.nl # IP:
>>>>>>>>> 192.0.2.10 # Storage used: 1.5T # Full backups to
>>>>>>>>> keep: 4 # Create full backup if last full backup is
>>>>>>>>> older than: 14D 
>>>>>>>>> ----------------------------------------- #
>>>>>>>>> Duplicity collection status: # Local and Remote
>>>>>>>>> metadata are synchronized, no sync needed. # Last
>>>>>>>>> full backup date: Mon May 25 10:48:16 2015 #
>>>>>>>>> Collection Status # ----------------- # Connecting
>>>>>>>>> with backend: BackendWrapper # Archive dir: 
>>>>>>>>> /root/.cache/duplicity/vps1.sparklingclouds.nl. #
>>>>>>>>> Found 4 secondary backup chains. # Secondary chain
>>>>>>>>> 1 of 4: # ------------------------- # Chain start
>>>>>>>>> time: Tue Mar 24 14:42:44 2015 # Chain end time:
>>>>>>>>> Tue Mar 24 14:42:44 2015 # Number of contained
>>>>>>>>> backup sets: 1 # Total number of contained volumes:
>>>>>>>>> 3009 #  Type of backup set: Time: Num volumes: #
>>>>>>>>> Full Tue Mar 24 14:42:44 2015 3009 # 
>>>>>>>>> ------------------------- # Secondary chain 2 of 4:
>>>>>>>>> # ------------------------- # Chain start time: Wed
>>>>>>>>> Apr 8 05:49:18 2015 # Chain end time: Wed Apr  8
>>>>>>>>> 05:49:18 2015 # Number of contained backup sets: 1
>>>>>>>>> # Total number of contained volumes: 3104 #  Type
>>>>>>>>> of backup set: Time:      Num volumes: #
>>>>>>>>> Full Wed Apr 8 05:49:18 2015 3104 # 
>>>>>>>>> ------------------------- # Secondary chain 3 of 4:
>>>>>>>>> # ------------------------- # Chain start time: Thu
>>>>>>>>> Apr 16 05:49:37 2015 # Chain end time: Thu Apr 16
>>>>>>>>> 05:49:37 2015 # Number of contained backup sets: 1
>>>>>>>>> # Total number of contained volumes: 3128 #  Type
>>>>>>>>> of backup set: Time:      Num volumes: #
>>>>>>>>> Full Thu Apr 16 05:49:37 2015 3128 #
>>>>>>>>> ------------------------- # Secondary chain 4 of 4:
>>>>>>>>> # ------------------------- # Chain start time: Thu
>>>>>>>>> Apr 30 05:49:39 2015 # Chain end time: Thu Apr 30
>>>>>>>>> 05:49:39 2015 # Number of contained backup sets: 1
>>>>>>>>> # Total number of contained volumes: 736 # Type of
>>>>>>>>> backup set: Time:      Num volumes: # Full Thu Apr
>>>>>>>>> 30 05:49:39 2015 736 # ------------------------- #
>>>>>>>>> Found primary backup chain with matching signature
>>>>>>>>> chain: # ------------------------- # Chain start
>>>>>>>>> time: Mon May 25 10:48:16 2015 # Chain end time:
>>>>>>>>> Mon May 25 10:48:16 2015 # Number of contained
>>>>>>>>> backup sets: 1 # Total number of contained volumes:
>>>>>>>>> 0 #  Type of backup set: Time:      Num volumes: #
>>>>>>>>> ------------------------- # No orphaned or
>>>>>>>>> incomplete backup sets found.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Why does this happen and how can I get this back to
>>>>>>>>>  normal?
>>>>>>>> 
>>>>>> 
>>>>>>> which duplicity version?
>>>>>> 
>>>>>>> please post (zip attach)or pastebin the complete
>>>>>>> console output of a max.verbosity '-v9' status run.
>>>>>>> obfuscate strings in it you deem private.
>>>>>> 
>>>>>>> ..ede/duply.net
>>>>>> 
>>>> 
>>>>> the log states
>>>> 
>>>>> --> DEBUG 1 . Found backup chain [Tue Mar 24 14:42:44 
>>>>> 2015]-[Tue Mar 24 14:42:44 2015]
>>>> 
>>>>> DEBUG 1 . Found backup chain [Wed Apr  8 05:49:18
>>>>> 2015]-[Wed Apr  8 05:49:18 2015]
>>>> 
>>>>> DEBUG 1 . Found backup chain [Thu Apr 16 05:49:37
>>>>> 2015]-[Thu Apr 16 05:49:37 2015]
>>>> 
>>>>> DEBUG 1 . Found backup chain [Thu Apr 30 05:49:39
>>>>> 2015]-[Thu Apr 30 05:49:39 2015]
>>>> 
>>>>> NOTICE 1 . Last full backup date: Thu Apr 30 05:49:39 2015
>>>> 
>>>>> NOTICE 1 . Last full backup is too old, forcing full
>>>>> backup <--
>>>> 
>>>>> which sounds reasonable.. there are no incremental backups
>>>>> on your backend.
>>>> 
>>>> There should be. There runs a cronjob every night with the
>>>> same command as I ran manually. It runs for about 3 to 5
>>>> hours just as now, and then fails. The backend keeps growing
>>>> in size.
>>>> 
>>>> 
>> 
>>> ok, please ensure that your manual full went through via
>>> 'status' command.
>> 
>> 
>> It is not listed in the output of collection-status. Should I
>> still run the backup again?
>> 
>>> then rerun the same backup cmd line you sent me before and send
>>> me the output up until the first 'Getting delta of ...'
>>> message.
>> 
> 
> weird, can you send a list of the files on your backend? looks like
> not all were uploaded successfully.
> 

Here's a list of all the files in the backend:

https://3335ae642c4a42549b7a4489adf98d7c.objectstore.eu/duplicity/duplic
ity-3.txt

> next step would be to try latest duplicity 0.6.x . maybe some bug
> crept in the 0.7 development line. here is a short howto install
> from tarball to a local folder (under INSTALL MULTIPLE VERSIONS). 
> http://duply.net/wiki/index.php/Duply-documentation you might wanna
> to install to /usr/local/duplicity-0.6.x instead.

I'll try it with the latest 0.6 series as well.

> 
> ..ede/duply.net
> 
> _______________________________________________ Duplicity-talk
> mailing list address@hidden 
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCAAGBQJVaEuAAAoJECtnVb0bf4jc3pEIAKxRIWn1ARrpzqBYAqgE11gO
3t6+aUw6rpJ6RveNA6nl+WbBj9JaoWnrGLgN/4GKq4CJRjjQDIW1X4FBPrVfNWZN
2cmLqlMaBYKSkEKX5pzRH0Bse8xZ1pjW4APN5ajLfPPiC8AxmTAz8VwQJNZOW2Pj
hE+oJFhQrdsJsAdq+brFRQB15B+O4enAQDoNocIlqqDO4oEQC7FQtn0eSu12Qkl9
mShVGn0mYMeYDP7CswlwUYTu5yrWNleJ2EhK+uFFNsYya5RtscPpHRrY+Pg7gHBM
077ENO83UVzEhta5bkSMgavI7HtZnL3JxR1hqDFFax3/J9JER5RBrCSDD5aGGDA=
=8Ieo
-----END PGP SIGNATURE-----

0x1B7F88DC.asc
Description: application/pgp-keys

0x1B7F88DC.asc.sig
Description: PGP signature