Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?

[edgar . soldin]

please do not top post if the answer was posted at the bottom.. i will continue 
below

On 28.05.2015 19:47, Remy van Elst wrote:
> Here's the first 31000 lines of the log:
> https://e75f5515c4f249ffa1122853af811b3e.objectstore.eu/duplicity/duplic
> ity.txt
> 
> After these, it is all lines comparing the files on the machine. That
> is still running. Is this helpfull?
> 
> On 05/28/2015 03:54 PM, address@hidden wrote:
>> On 28.05.2015 15:28, Remy van Elst wrote:
>>> Any suggestions or other help?
>>>
>>> On 05/25/2015 11:04 AM, Remy van Elst wrote:
>>>> Duplicity seems to loose it's status when trying to do a 
>>>> incremental backup set over 14 days. The backup has been
>>>> running for a while now but it is using about 1.5 TB of backend
>>>> storage since 2 weeks. It is going to an Openstack swift object
>>>> store as backend.
>>>
>>>> This is the command line:
>>>
>>>> duplicity --asynchronous-upload --verbosity 9 --log-file 
>>>> /var/log/duplicity.log --volsize 25 --tempdir="/tmp" 
>>>> --file-prefix="vps1.sparklingclouds.nl." 
>>>> --name="vps1.sparklingclouds.nl." --exclude-device-files 
>>>> --exclude-globbing-filelist=/etc/duplicity-backup/exclude.conf
>>>>  --full-if-older-than="14D" --no-encryption  / 
>>>> swift://duplicity-backup
>>>
>>>> This is what caught my eye in the verbose log file:
>>>
>>>> File 
>>>> vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol166.difft
> ar
>>>
>>>>
> .g
>>>
>>>
>>> z
>>>> is part of known set File 
>>>> vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol1660.diff
> ta
>>>
>>>>
> r.
>>>
>>>
>>> gz
>>>> is part of known set Found backup chain [Tue Mar 24 14:42:44 
>>>> 2015]-[Tue Mar 24 14:42:44 2015] Last full backup is too old, 
>>>> forcing full backup
>>>
>>>
>>>> The last full backup is from yesterday, and it should only do
>>>> a full backup every 14 days.
>>>
>>>> When I do a collection status it seems all the incremental
>>>> backups are gone:
>>>
>>>> Do note that I've got a wrapper script around duplicity which 
>>>> outputs the status, storage used and config settings:
>>>
>>>
>>>> # Hostname: vps1.sparklingclouds.nl # IP: 192.0.2.10 # Storage 
>>>> used: 1.5T # Full backups to keep: 4 # Create full backup if
>>>> last full backup is older than: 14D 
>>>> ----------------------------------------- # Duplicity
>>>> collection status: # Local and Remote metadata are
>>>> synchronized, no sync needed. # Last full backup date: Mon May
>>>> 25 10:48:16 2015 # Collection Status # ----------------- #
>>>> Connecting with backend: BackendWrapper # Archive dir: 
>>>> /root/.cache/duplicity/vps1.sparklingclouds.nl. # Found 4
>>>> secondary backup chains. # Secondary chain 1 of 4: # 
>>>> ------------------------- # Chain start time: Tue Mar 24
>>>> 14:42:44 2015 # Chain end time: Tue Mar 24 14:42:44 2015 #
>>>> Number of contained backup sets: 1 # Total number of contained
>>>> volumes: 3009 #  Type of backup set:
>>>> Time:      Num volumes: #                 Full         Tue Mar
>>>> 24 14:42:44 2015 3009 # ------------------------- # Secondary
>>>> chain 2 of 4: # ------------------------- # Chain start time:
>>>> Wed Apr  8 05:49:18 2015 # Chain end time: Wed Apr  8 05:49:18
>>>> 2015 # Number of contained backup sets: 1 # Total number of
>>>> contained volumes: 3104 #  Type of backup set:
>>>> Time:      Num volumes: #                 Full         Wed Apr
>>>> 8 05:49:18 2015 3104 # ------------------------- # Secondary
>>>> chain 3 of 4: # ------------------------- # Chain start time:
>>>> Thu Apr 16 05:49:37 2015 # Chain end time: Thu Apr 16 05:49:37
>>>> 2015 # Number of contained backup sets: 1 # Total number of
>>>> contained volumes: 3128 #  Type of backup set:
>>>> Time:      Num volumes: #                 Full         Thu Apr
>>>> 16 05:49:37 2015 3128 # ------------------------- # Secondary
>>>> chain 4 of 4: # ------------------------- # Chain start time:
>>>> Thu Apr 30 05:49:39 2015 # Chain end time: Thu Apr 30 05:49:39
>>>> 2015 # Number of contained backup sets: 1 # Total number of
>>>> contained volumes: 736 # Type of backup set:
>>>> Time:      Num volumes: #                 Full         Thu Apr
>>>> 30 05:49:39 2015 736 # ------------------------- # Found
>>>> primary backup chain with matching signature chain: #
>>>> ------------------------- # Chain start time: Mon May 25
>>>> 10:48:16 2015 # Chain end time: Mon May 25 10:48:16 2015 #
>>>> Number of contained backup sets: 1 # Total number of contained
>>>> volumes: 0 #  Type of backup set: Time:      Num volumes: #
>>>> ------------------------- # No orphaned or incomplete backup
>>>> sets found.
>>>
>>>
>>>> Why does this happen and how can I get this back to normal?
>>>
> 
>> which duplicity version?
> 
>> please post (zip attach)or pastebin the complete console output of
>> a max.verbosity '-v9' status run. obfuscate strings in it you deem
>> private.
> 
>> ..ede/duply.net
> 

the log states

-->
DEBUG 1
. Found backup chain [Tue Mar 24 14:42:44 2015]-[Tue Mar 24 14:42:44 2015]

DEBUG 1
. Found backup chain [Wed Apr  8 05:49:18 2015]-[Wed Apr  8 05:49:18 2015]

DEBUG 1
. Found backup chain [Thu Apr 16 05:49:37 2015]-[Thu Apr 16 05:49:37 2015]

DEBUG 1
. Found backup chain [Thu Apr 30 05:49:39 2015]-[Thu Apr 30 05:49:39 2015]

NOTICE 1
. Last full backup date: Thu Apr 30 05:49:39 2015

NOTICE 1
. Last full backup is too old, forcing full backup
<--

which sounds reasonable.. there are no incremental backups on your backend. 

1. how long does a full backup run on your box?
2. how often is the backup job started via cron?

..ede/duply.net