Re: [Duplicity-talk] why restart takes so long

[covici]

I have thousands of files, some large and some small, its about a
terabytes worth.  It stopped pretty near the end and it just stopped
again, so my emphasis wuld be to fix the error and maybe do something
about optimizing the program.


Kenneth Loafman <address@hidden> wrote:

> Do you have a lot of very small files?  This should not take too long.
> 
> 
> On Tue, May 10, 2016 at 12:02 PM, <address@hidden> wrote:
> 
> > Well, what happens is that it says A and all the filenames, but no
> > reference to a tar file, so its not really backing up again, this is why
> > I was wondering what it is doing.  The part of the program which seems
> > to be involved is this:
> > def restart_position_iterator(tarblock_iter):
> >     """
> >         Fake writing to backend, but do go through all the source paths.
> >                Stop when we have processed the last file and block from the
> >                       last backup.  Normal backup will proceed at the
> > start of
> >                          the
> >                              next volume in the set.
> >
> >     @type tarblock_iter: tarblock_iter
> >         @param tarblock_iter: iterator for current tar block
> >
> >     @rtype: int
> >         @return: constant 0 (zero)
> >                """
> >                       last_index = globals.restart.last_index
> >                              last_block = globals.restart.last_block
> >                                     try:
> >                                             # Just spin our wheels
> >                                                           iter_result =
> >         tarblock_iter.next()
> >                    while iter_result:
> >                                      if (tarblock_iter.previous_index ==
> >         last_index):
> >                            # If both the previous index and this index are
> >         done, exit now
> >                            # before we hit the next index, to prevent
> >         skipping its first
> >                            # block.
> >                                                     if not last_block and
> > not
> >         tarblock_iter.previous_block:
> >                                break
> >                                                            # Only check
> > block number if
> >         last_block is also a number
> >                            if last_block and tarblock_iter.previous_block >
> >         last_block:
> >                                break
> >                                                        if
> > tarblock_iter.previous_index
> >         > last_index:
> >                            log.Warn(_("File %s complete in backup set.\n"
> >                                                                "Continuing
> > restart
> >         on file %s.") %
> >
> > Kenneth Loafman <address@hidden> wrote:
> >
> > > It sounds like the backup restarted completely, rather than at the last
> > > known spot.  What do the first 50 or so lines of the log contain?
> > >
> > > ...Ken
> > >
> > >
> > > On Tue, May 10, 2016 at 10:48 AM, <address@hidden> wrote:
> > >
> > > > But the  question is why does duplicity need to do that at all?
> > > > Couldn't it go through the last hundred or so or some reasonable
> > number?
> > > > Also, what about the original error which caused the restart in the
> > > > first place?  What is up with that that?
> > > >
> > > >
> > > > Kenneth Loafman <address@hidden> wrote:
> > > >
> > > > > No, there is no way to get rid of the loop, whichever one you're
> > talking
> > > > > about, there are a lot of them.
> > > > >
> > > > > The main suggestion I have for anyone with multi-hour backups is to:
> > > > >
> > > > >    1. Use a local filesystem or USB drive for backup,
> > > > >    2. Transfer the backup using rsync or similar tool.
> > > > >
> > > > > The second suggestion I have is to chunk the backup into reasonably
> > sized
> > > > > portions, 200-500GB each.
> > > > >
> > > > > ...Ken
> > > > >
> > > > >
> > > > > On Tue, May 10, 2016 at 7:02 AM, <address@hidden> wrote:
> > > > >
> > > > > > Hi.  I had to restart my full backup and its going through every
> > file
> > > > it
> > > > > > had and it is taking 18 hours and its not done yet.  I see a loop
> > in
> > > > > > there and I wonder if there is any way to shorten the time or get
> > rid
> > > > of
> > > > > > the loop altoggether?
> > > > > >
> > > > > > Thanks in advance for any suggestions.
> > > > > >
> > > > > > --
> > > > > > Your life is like a penny.  You're going to lose it.  The question
> > is:
> > > > > > How do
> > > > > > you spend it?
> > > > > >
> > > > > >          John Covici
> > > > > >          address@hidden
> > > > > >
> > > > > > _______________________________________________
> > > > > > Duplicity-talk mailing list
> > > > > > address@hidden
> > > > > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > > > >
> > > > >
> > > > > ----------------------------------------------------
> > > > > Alternatives:
> > > > >
> > > > > ----------------------------------------------------
> > > > > _______________________________________________
> > > > > Duplicity-talk mailing list
> > > > > address@hidden
> > > > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > >
> > > > --
> > > > Your life is like a penny.  You're going to lose it.  The question is:
> > > > How do
> > > > you spend it?
> > > >
> > > >          John Covici
> > > >          address@hidden
> > > >
> > > > _______________________________________________
> > > > Duplicity-talk mailing list
> > > > address@hidden
> > > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > >
> > >
> > > ----------------------------------------------------
> > > Alternatives:
> > >
> > > ----------------------------------------------------
> > > _______________________________________________
> > > Duplicity-talk mailing list
> > > address@hidden
> > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >
> > --
> > Your life is like a penny.  You're going to lose it.  The question is:
> > How do
> > you spend it?
> >
> >          John Covici
> >          address@hidden
> >
> > _______________________________________________
> > Duplicity-talk mailing list
> > address@hidden
> > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >
> 
> ----------------------------------------------------
> Alternatives:
> 
> ----------------------------------------------------
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         address@hidden