[rdiff-backup-users] excessive fsync(2)

[Greg Troxel]

With 1.2.8 on NetBSD, I was seeing insanely long backup times, as in
over a week for a 300G filesystem on which little had changed.
With ktrace, I found the problem was that fsync() was called incredibly
often (per file checked?) and my backup drive was mounted with WAPBL
(metadata journaling), and fsync causes the log to be written, which
causes a cache flush on the drive.  fsync was taking 3/4s most of the
time.

I rebuilt rdiff-backup with the fsync call commented out and then it ran
reasonably quickly).  On another machine with two differences: disk not
mounted with wapbl and rdiff-backup client remote and accessing it over
ssh, either of which might matter, backups go reasonably quickly as
well.

One can argue that this implementation of fsync (cache flush, rather
than some sort of barrier) is inefficient, but fsync cannot properly
return until all the file's data and metadata is safely on disk, so it's
intrinscially a non-streaming expensive operation.

Can anyone explain why rdiff-backup is doing fsyncs so often?

I would expect the plan to be to write all the files and record the
files-written log, and then at the end sync(2) it all (and then wait,
since sync doesn't wait to completion), and then to remove the
backup-in-progress flag.  with the regress-dirty strategy, that kind of
makes the whole thing a transaction.

Thanks,
Greg

pgpFwUqasrthJ.pgp
Description: PGP signature