[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [rdiff-backup-users] Delay between EndTime and process end - followu
Re: [rdiff-backup-users] Delay between EndTime and process end - followup
Mon, 12 Aug 2013 08:57:45 -0500
Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130807 Thunderbird/17.0.8
On 08/12/2013 07:09 AM, Laurent De Buyst wrote:
2013/08/12 06:00:56 Mon Aug 12 05:56:03 2013 Copying inc attrs from ... to
2013/08/12 06:00:56 Mon Aug 12 05:56:03 2013 Setting time of ... to
2013/08/12 06:00:56 Mon Aug 12 05:57:36 2013 Renaming ... to
2013/08/12 06:00:56 Mon Aug 12 06:00:56 2013 Setting time of .../backup/tmp
2013/08/12 06:00:56 Mon Aug 12 06:00:56 2013 Copying attributes from () to
The batching of messages is almost certainly due to stdio buffering of
the output stream. Output that is going to a non-terminal device will,
by default, be collected in a buffer that is typically 4096 bytes and
only written out when the buffer becomes full. You are probably piping
the output into your wrapper script, so stdio just sees the pipe and
uses block buffering. There are utilities like "unbuffer" (part of the
"expect" package) that will interpose a pty device in the stream so that
stdio will see a terminal device and use line buffering. But, this is
not related to your larger delay problem.
It sounds like rdiff-backup is issuing a sync() call to ensure that all
data has been flushed to disk prior to writing its final log messages.
That sync() call affects the whole system, so if you have many gigabytes
of data in kernel buffers and multiple processes still actively writing,
it could take quite a while for that call to return. I'm not sure just
how the kernel deals with buffers that are still being written while the
sync()-ing is being done, but I can see the possibility that it is
waiting for _all_ the buffers to become clean before returning, and
there is no certainty of when that will occur in a busy system. You
might inquire on the Linux kernel mailing list about this behavior.
Paradoxically, you might be better off with _less_ memory in the server
(less memory for buffers). That's probably not a practical solution.
Bob Nichols "NOSPAM" is really part of my email address.
Do NOT delete it.