Re: [Gnu-arch-users] Re: Backing up Arch archives

[Tom Lord]

    > From: address@hidden (Julian T. J. Midgley)

    > [....] I vaguely remember reading somewhere that it manipulates the
    > repository by creating temporary directories, filling them with
    > whatever stuff is required, and renaming them into place when
    > finished.  If so, presumably the worst that could occur if, say,
    > someone committed part way through a backup, is that the backup
    > ends up containing a random temporary directory that tla would
    > presumably ignore if you ever had to restore from it.

Close.  

The problem is that a program like `tar' is going to non-atomicly
traverse the tree.  That means that at time T it's going to "take a
snapshot" of some part of the archive tree and at time T+1 it'll take
a snapshot of some other part of the archive tree.  tar makes no
guarantee about the ordering of scope of those snapshots.

The locking protocols of arch guarantee that at every instant in time
the lock-dirs are in a sane state --- they do _not_ guarantee that if
you splice-together to partial snapshots of an archive, taken at
different times, that the result has the lock in a sane state.

It _is_ guaranteed that if you just tar off a live archive that the
result will contain no invalid revisions --- if a roughly concurrent
commit is happening, either you'll get that revision in its entirety
or you won't get it at all.  (And, yes, you may get some temp dirs.).

However, your results might be:

1) tar failing because it thought a directory was there but tla
   concurrently removed it just as tar had committed to opendir'ing
   it.   Hopefully tar will give you only a warning here (which you
   can ignore) but, if not, just retry.

2) Extra junk dirs that tla will, indeed, just ignore.  You can delete
   these freely.

3) A locked version.  That is, if you restore from this backup and try
   to commit to a certain version, tla will tell you that the lock is
   held.   You just use `tla lock-revision --break' to fix that.
   It's the same as if somebody had started a commit but interrupted
   it before it was complete (but after acquiring the lock).

4) A version that is missing its lock-dir.    You can not commit
   to this version until you create, by hand, a new lockdir (which
   takes exactly 3 shell commands: two mkdir's and a rename).

   You may wonder why arch doesn't contain a command to recreate a
   missing lockdir.   Two reasons.   First, it can't happen except in
   cases like this use of tar.   Second, recreating a missing lockdir
   can not be done safely if there is a possibility that two users
   might try to do it at the same time (and concurrently with a third
   user attempting a commit).

The upshot of this is that, for a completely automated backup process
that will _never_ require any hand-intervention to replace a missing
lockdir, you must have, at some point, a quiescent archive.   But not
necessarily when you think:

Please note carefully -- it doesn't _have_ to be quiescent at backup
time.  It's sufficient for it to be quiescent at restoration time.
The pattern would be something like:


        1) Backup a live archive using tar, as above.

        2) Oh, no, disaster struck and you need to restore from tape.

        3) Restore the archive tree but, before putting it on-line,
           replace any missing lockdirs.

And, hmm.... while (3) can be done with a trivial shell-script,
probably arch should have a command for that after all, just to make
the process stone-simple.



    > Is this true for all operations that can be performed an archive?
    > I.e. can one take self-consistent backups of live Arch archives with
    > simple recursive copy operations?  Apologies if this has been asked a
    > thousand times before, or if it's obvious(ly wrong?); I'll summarise
    > the responses of the knowledgeable on the Wiki so it doesn't crop up
    > again.

Yes.... live backups are perfectly safe.  You only need to add back
any missing lockdirs after restoring from backup.    Should you forget
to do that, eventually you'll realize that the lock in question is
missing and you can create it then.

-t