Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?

[edgar . soldin]

On 29.05.2015 04:41, Remy van Elst wrote:
> 
> 
> On 05/29/2015 12:25 AM, address@hidden wrote:
>> please do not top post if the answer was posted at the bottom.. i
>> will continue below
> 
>> On 28.05.2015 19:47, Remy van Elst wrote:
>>> Here's the first 31000 lines of the log: 
>>> https://e75f5515c4f249ffa1122853af811b3e.objectstore.eu/duplicity/dup
> lic
>>>
>>>
> ity.txt
>>>
>>> After these, it is all lines comparing the files on the machine.
>>> That is still running. Is this helpfull?
>>>
>>> On 05/28/2015 03:54 PM, address@hidden wrote:
>>>> On 28.05.2015 15:28, Remy van Elst wrote:
>>>>> Any suggestions or other help?
>>>>>
>>>>> On 05/25/2015 11:04 AM, Remy van Elst wrote:
>>>>>> Duplicity seems to loose it's status when trying to do a 
>>>>>> incremental backup set over 14 days. The backup has been 
>>>>>> running for a while now but it is using about 1.5 TB of
>>>>>> backend storage since 2 weeks. It is going to an Openstack
>>>>>> swift object store as backend.
>>>>>
>>>>>> This is the command line:
>>>>>
>>>>>> duplicity --asynchronous-upload --verbosity 9 --log-file 
>>>>>> /var/log/duplicity.log --volsize 25 --tempdir="/tmp" 
>>>>>> --file-prefix="vps1.sparklingclouds.nl." 
>>>>>> --name="vps1.sparklingclouds.nl." --exclude-device-files 
>>>>>> --exclude-globbing-filelist=/etc/duplicity-backup/exclude.conf
>>>>>>
>>>>>>
> --full-if-older-than="14D" --no-encryption  /
>>>>>> swift://duplicity-backup
>>>>>
>>>>>> This is what caught my eye in the verbose log file:
>>>>>
>>>>>> File 
>>>>>> vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol166.dif
> ft
>>>
>>>>>>
> ar
>>>>>
>>>>>>
>>> .g
>>>>>
>>>>>
>>>>> z
>>>>>> is part of known set File 
>>>>>> vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol1660.di
> ff
>>>
>>>>>>
> ta
>>>>>
>>>>>>
>>> r.
>>>>>
>>>>>
>>>>> gz
>>>>>> is part of known set Found backup chain [Tue Mar 24
>>>>>> 14:42:44 2015]-[Tue Mar 24 14:42:44 2015] Last full backup
>>>>>> is too old, forcing full backup
>>>>>
>>>>>
>>>>>> The last full backup is from yesterday, and it should only
>>>>>> do a full backup every 14 days.
>>>>>
>>>>>> When I do a collection status it seems all the incremental 
>>>>>> backups are gone:
>>>>>
>>>>>> Do note that I've got a wrapper script around duplicity
>>>>>> which outputs the status, storage used and config
>>>>>> settings:
>>>>>
>>>>>
>>>>>> # Hostname: vps1.sparklingclouds.nl # IP: 192.0.2.10 #
>>>>>> Storage used: 1.5T # Full backups to keep: 4 # Create full
>>>>>> backup if last full backup is older than: 14D 
>>>>>> ----------------------------------------- # Duplicity 
>>>>>> collection status: # Local and Remote metadata are 
>>>>>> synchronized, no sync needed. # Last full backup date: Mon
>>>>>> May 25 10:48:16 2015 # Collection Status #
>>>>>> ----------------- # Connecting with backend: BackendWrapper
>>>>>> # Archive dir: 
>>>>>> /root/.cache/duplicity/vps1.sparklingclouds.nl. # Found 4 
>>>>>> secondary backup chains. # Secondary chain 1 of 4: # 
>>>>>> ------------------------- # Chain start time: Tue Mar 24 
>>>>>> 14:42:44 2015 # Chain end time: Tue Mar 24 14:42:44 2015 # 
>>>>>> Number of contained backup sets: 1 # Total number of
>>>>>> contained volumes: 3009 #  Type of backup set: Time:
>>>>>> Num volumes: #                 Full         Tue Mar 24
>>>>>> 14:42:44 2015 3009 # ------------------------- # Secondary 
>>>>>> chain 2 of 4: # ------------------------- # Chain start
>>>>>> time: Wed Apr  8 05:49:18 2015 # Chain end time: Wed Apr  8
>>>>>> 05:49:18 2015 # Number of contained backup sets: 1 # Total
>>>>>> number of contained volumes: 3104 #  Type of backup set: 
>>>>>> Time:      Num volumes: #                 Full         Wed
>>>>>> Apr 8 05:49:18 2015 3104 # ------------------------- #
>>>>>> Secondary chain 3 of 4: # ------------------------- # Chain
>>>>>> start time: Thu Apr 16 05:49:37 2015 # Chain end time: Thu
>>>>>> Apr 16 05:49:37 2015 # Number of contained backup sets: 1 #
>>>>>> Total number of contained volumes: 3128 #  Type of backup
>>>>>> set: Time:      Num volumes: #                 Full
>>>>>> Thu Apr 16 05:49:37 2015 3128 # ------------------------- #
>>>>>> Secondary chain 4 of 4: # ------------------------- # Chain
>>>>>> start time: Thu Apr 30 05:49:39 2015 # Chain end time: Thu
>>>>>> Apr 30 05:49:39 2015 # Number of contained backup sets: 1 #
>>>>>> Total number of contained volumes: 736 # Type of backup
>>>>>> set: Time:      Num volumes: #                 Full
>>>>>> Thu Apr 30 05:49:39 2015 736 # ------------------------- #
>>>>>> Found primary backup chain with matching signature chain:
>>>>>> # ------------------------- # Chain start time: Mon May 25 
>>>>>> 10:48:16 2015 # Chain end time: Mon May 25 10:48:16 2015 # 
>>>>>> Number of contained backup sets: 1 # Total number of
>>>>>> contained volumes: 0 #  Type of backup set: Time:      Num
>>>>>> volumes: # ------------------------- # No orphaned or
>>>>>> incomplete backup sets found.
>>>>>
>>>>>
>>>>>> Why does this happen and how can I get this back to
>>>>>> normal?
>>>>>
>>>
>>>> which duplicity version?
>>>
>>>> please post (zip attach)or pastebin the complete console output
>>>> of a max.verbosity '-v9' status run. obfuscate strings in it
>>>> you deem private.
>>>
>>>> ..ede/duply.net
>>>
> 
>> the log states
> 
>> --> DEBUG 1 . Found backup chain [Tue Mar 24 14:42:44 2015]-[Tue
>> Mar 24 14:42:44 2015]
> 
>> DEBUG 1 . Found backup chain [Wed Apr  8 05:49:18 2015]-[Wed Apr  8
>> 05:49:18 2015]
> 
>> DEBUG 1 . Found backup chain [Thu Apr 16 05:49:37 2015]-[Thu Apr 16
>> 05:49:37 2015]
> 
>> DEBUG 1 . Found backup chain [Thu Apr 30 05:49:39 2015]-[Thu Apr 30
>> 05:49:39 2015]
> 
>> NOTICE 1 . Last full backup date: Thu Apr 30 05:49:39 2015
> 
>> NOTICE 1 . Last full backup is too old, forcing full backup <--
> 
>> which sounds reasonable.. there are no incremental backups on your
>> backend.
> 
> There should be. There runs a cronjob every night with the same
> command as I ran manually. It runs for about 3 to 5 hours just as now,
> and then fails. The backend keeps growing in size.
> 
> 

ok, please ensure that your manual full went through via 'status' command.

then rerun the same backup cmd line you sent me before and send me the output 
up until the first 'Getting delta of ...' message.

..ede/duply.net