Hi,
Yes that's correct. A full backup was interrupted, files changed, backup resumed. Then I ran an incremental backup instantly that apparently didn't pick up changes.
"duplicity restore"-ing the backup chain results in files from the first attempt (so neither the resumed full backup nor the incremental picked up changed files)
duplicity verify:
> Verify complete: 68 files compared, 0 differences found.
duplicity verify --compare-data
> Verify complete: 68 files compared, 68 differences found.
thanks,
Mark
On Mon, Jul 23, 2018 at 1:12 PM Kenneth Loafman <
address@hidden> wrote:
Hi,
Looks like an interesting problem!
To summarize:
- Full backup started
- Full backup interrupted
- Files changed that were already backed up
- Full backup resumed and completed
- Inc backup does not pick up files changed
At this point, a verify should fail for the files in #3 (sigtar has new files, difftar has old files maybe). Would you please run a verify and confirm? Run the verify without --compare-data.
...Thanks,
...Ken
Hi,
Yes, the full backup was interrupted (it crashed because it couldn't upload the >2GB sigtar archive). Then (about 24 hours later) I added max-blocksize and re-ran it, then it completed the full backup with results explained below.
Unfortunately I don't know Python so I can't help with implementation but I am happy to discuss / test / verify or assist in an other way.
Mark
hey Mark,
On 23.07.2018 11:33, Mark Diab wrote:
> Hi,
>
> In the meantime I have tested it and it indeed seems to be the case :(
>
> I ran an incremental backup right after the full backup, it didn't detect changes. Then I tried to restore some files that were changed between crash<>resume and while "duplicity list" shows the proper (new) dates/times, duplicity restore restored old (at time of crash) files. What's worse, it now thinks that those new files are backed up so it won't ever make incremental backups until a new full backup is started.
just for clarification, which backup was interrupted and resumed. the full right?
> It means that it leaves an invisible "hole" in backups that is very tricky to fix or detect. Files created in the interim period will not be saved unless they are changed again.
>
> This could be fixed by either processing the previous (crashed) snapshot file at recovery or simply NOT ignoring any files that were changed since the first backup attempt (when recovery is fast forwarding to the last backed up file). I am not sure how the system reacts if a backup archive has multiple copies of the same file though so this may make implementation difficult.
yeah, resuming is a tricky beast, let's see what Ken has to say about. in the meanwhile, if you feel capable a patch to fix the issue would be highly appreciated.
..ede/duply.net
> thanks,
>
> Mark
>
>
>
>
> On Mon, Jul 23, 2018 at 11:16 AM <address@hidden <mailto:address@hidden>> wrote:
>
> as Ken implemented that feature i guess he can come back to you w/ implementation details.
>
> ..ede/duply.net <http://duply.net>
>
> On 23.07.2018 08:54, Mark Diab via Duplicity-talk wrote:
> > Hi List,
> >
> > I'm running duplicity (0.7.10) to back up several servers and recently I've hit the signature file >2GB bug when incrementals became too old and it tried to make a new full backup, so I had to increase --max-blocksize.
> >
> > Duplicity seems to be recovering OK, however, it made me wonder about files that have changed in the meantime (between the crash and the resumed recovery). It's quite a lot of files so I'd prefer not to start over from the beginning of the full backup but resume what's already uploaded.
> >
> > Now, if i understand correctly resume is basically implemented as a "fake" backup not doing any uploads up to the point of the last backed up file. I can see that the SigTar file is regenerated. I assume that sigtar contains the metadata to detect changes. However, this may lead to a situation where checksums / file meta information contains different data (time of resume) than the actual archive (backed before those files were changed).
> >
> > My question is if anyone knows what happens to those files? Is this how resume works? Does it mean that duplicity is never going to back up files that were changes between crash<>resume because their change went unnoticed since the meta data contains newer files than the archive uploaded before the crash?
> >
> > thanks,
> >
> > Mark
> >
> >
> >
> >
> > _______________________________________________
> > Duplicity-talk mailing list
> > address@hidden <mailto:address@hidden>
> > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >
>