Re: [Duplicity-talk] Having to specify incrementals

[edgar . soldin]

On 04.04.2013 18:10, Elvar wrote:
> 
> On 4/4/2013 11:05 AM, address@hidden wrote:
>> On 04.04.2013 18:02, Elvar wrote:
>>> On 4/4/2013 10:24 AM, address@hidden wrote:
>>>> On 04.04.2013 17:16, Elvar wrote:
>>>>> On 3/29/2013 5:29 PM, address@hidden wrote:
>>>>>> On 29.03.2013 22:24, Elvar wrote:
>>>>>>> On 3/29/2013 1:59 PM, address@hidden wrote:
>>>>>>>> On 29.03.2013 19:35, Elvar wrote:
>>>>>>>>> On 3/29/2013 11:49 AM, address@hidden wrote:
>>>>>>>>>> On 29.03.2013 17:31, Elvar wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I recently started using Duplicity to perform offsite backups for a 
>>>>>>>>>>> linux server of ours. When I performed  a simulated disaster 
>>>>>>>>>>> recovery scenario I found that the only data I had been able 
>>>>>>>>>>> restore was data from the initial full backup. It doesn't appear 
>>>>>>>>>>> that Duplicity was automatically doing incrementals despite the 
>>>>>>>>>>> data growing. Below is the command I'm using...
>>>>>>>>>>>
>>>>>>>>>>> FTP_PASSWORD='somepass' PASSPHRASE='somepass' duplicity /mnt 
>>>>>>>>>>> ftps://address@hidden
>>>>>>>>>>>
>>>>>>>>>>> Shouldn't that automatically assume incrementals if the full had 
>>>>>>>>>>> already been done?
>>>>>>>>>>>
>>>>>>>>>> yes. what's the output of collection-status?
>>>>>>>>>>
>>>>>>>>>> ..ede/duply.net
>>>>>>>>> After having manually ran an incremental or two, here is the current 
>>>>>>>>> status.
>>>>>>>>>
>>>>>>>>> Import of duplicity.backends.giobackend Failed: No module named gio
>>>>>>>>> Import of duplicity.backends.sshbackend Failed: No module named 
>>>>>>>>> paramiko
>>>>>>>>> LFTP version is 4.3.3
>>>>>>>>> Local and Remote metadata are synchronized, no sync needed.
>>>>>>>>> Last full backup date: Thu Mar 28 17:03:26 2013
>>>>>>>>> Collection Status
>>>>>>>>> -----------------
>>>>>>>>> Connecting with backend: FTPSBackend
>>>>>>>>> Archive dir: /root/.cache/duplicity/cb471964ea71f51bdbff729d2a8e763e
>>>>>>>>>
>>>>>>>>> Found 0 secondary backup chains.
>>>>>>>>>
>>>>>>>>> Found primary backup chain with matching signature chain:
>>>>>>>>> -------------------------
>>>>>>>>> Chain start time: Thu Mar 28 17:03:26 2013
>>>>>>>>> Chain end time: Fri Mar 29 11:23:11 2013
>>>>>>>>> Number of contained backup sets: 6
>>>>>>>>> Total number of contained volumes: 61
>>>>>>>>>      Type of backup set:                            Time:      Num 
>>>>>>>>> volumes:
>>>>>>>>>                     Full         Thu Mar 28 17:03:26 2013 50
>>>>>>>>>              Incremental         Thu Mar 28 19:24:47 2013             
>>>>>>>>>     1
>>>>>>>>>              Incremental         Fri Mar 29 09:23:59 2013             
>>>>>>>>>     1
>>>>>>>>>              Incremental         Fri Mar 29 09:33:08 2013             
>>>>>>>>>     1
>>>>>>>>>              Incremental         Fri Mar 29 10:12:58 2013             
>>>>>>>>>     1
>>>>>>>>>              Incremental         Fri Mar 29 11:23:11 2013             
>>>>>>>>>     7
>>>>>>>>> -------------------------
>>>>>>>>> No orphaned or incomplete backup sets found.
>>>>>>>>>
>>>>>>>> dunno what your complaining about. it clearly states
>>>>>>>>
>>>>>>>> 1 x full
>>>>>>>> 5 x incrementals
>>>>>>>>
>>>>>>>> run again without forced incremental and see if it adds full or incr.
>>>>>>>>
>>>>>>>> ..ede
>>>>>>>>
>>>>>>>>
>>>>>>> So I just ran it again and I see it says "NewFiles 0" and "NewFileSize 
>>>>>>> 0". The content I'm backing up is an email archive that uses Maildir 
>>>>>>> format for storage. I know for certain several hundred emails or more 
>>>>>>> have landed in this archive since my last incremental yet this last job 
>>>>>>> doesn't seem to see it. So, my question is, why isn't the incremental 
>>>>>>> job grabbing the latest data?
>>>>>>>
>>>>>>> Import of duplicity.backends.giobackend Failed: No module named gio
>>>>>>> Import of duplicity.backends.sshbackend Failed: No module named paramiko
>>>>>>> LFTP version is 4.3.3
>>>>>>> Local and Remote metadata are synchronized, no sync needed.
>>>>>>> Last full backup date: Thu Mar 28 17:03:26 2013
>>>>>>> --------------[ Backup Statistics ]--------------
>>>>>>> StartTime 1364591400.73 (Fri Mar 29 16:10:00 2013)
>>>>>>> EndTime 1364591418.34 (Fri Mar 29 16:10:18 2013)
>>>>>>> ElapsedTime 17.62 (17.62 seconds)
>>>>>>> SourceFiles 29069
>>>>>>> SourceFileSize 2409332518 (2.24 GB)
>>>>>>> NewFiles 0
>>>>>>> NewFileSize 0 (0 bytes)
>>>>>>> DeletedFiles 0
>>>>>>> ChangedFiles 0
>>>>>>> ChangedFileSize 0 (0 bytes)
>>>>>>> ChangedDeltaSize 0 (0 bytes)
>>>>>>> DeltaEntries 0
>>>>>>> RawDeltaSize 0 (0 bytes)
>>>>>>> TotalDestinationSizeChange 103 (103 bytes)
>>>>>>> Errors 0
>>>>>>> -------------------------------------------------
>>>>>>>
>>>>>> what happened to your first issue? is that solved?
>>>>>>
>>>>>> wrt. 0byte change . this can happen when software does not update 
>>>>>> timestamps correctly. but i doubt that's the case here.
>>>>>> how about simply restoring the latest archive and binary compare it to 
>>>>>> the current state? after that run 'verify' and see if that says there is 
>>>>>> one.
>>>>>>
>>>>>> ..ede/duply.net
>>>>>>
>>>>>>
>>>>> Ok, I think I may have figured out the issue. When I was running the 
>>>>> Duplicity backups initially I was doing it on a read only iscsi mounted 
>>>>> volume. I noticed if I mounted without read-only I didn't have the same 
>>>>> issue with it not backing up newer data. Is this expected behavior?
>>>>>
>>>> no, please check if timestamps are correctly changing when the iscsi 
>>>> device is mounted read only.
>>>>
>>>> ede/duply.net
>>>>
>>>>
>>> The timestamps of the data I'm backing up? Well, the data is almost 
>>> entirely archived email being stored in Maildir format. There is constantly 
>>> new files being created every few seconds. For whatever reason, when 
>>> mounted read-only, it skips the new files and no data changes. If I run the 
>>> same exact command mounting rw, it backs up all new files and modified 
>>> files since the last backup.
>>>
>> duplicity checks if timestamps changed to determine the need for backup. 
>> check for your maildir files if timestamps change over time when mounted ro. 
>> probably they don't and that causes the problem?
>>
>> ..ede
>>
> 
> The appliance which is doing the email archiving isn't mounted ro though so 
> it should be updating the time & date stamps as it performs writes. Wouldn't 
> a separate server mounting the same partition only in ro mode still see 
> changes?
> 
> 
>

generally yes, the timestamps should be updated. but i suppose they are not. 
please check manually if they are and report back.

..ede