[Monotone-devel] recent monotone server outages

[Nathaniel Smith]

Some of you may have noticed problems with pulling from the venge.net
monotone server in the last few days, either mysterious disconnections
or simple "connection refused".

The problem is that the server that hosts venge.net (among many other
things) apparently has a bad IDE controller, which was silently
corrupting data coming off the disk.  Outages were partly the server
refusing to deal with corrupt data, and partly things being taken down
to replace hardware, verify data, etc.

If you pulled from the monotone server during this period, don't worry
about your database having been corrupted as well; monotone's built-in
integrity checking ensures that it always gave out either good data,
or no data at all.  In fact, the problem was originally discovered
through the monotone server's complaints, and when replacing the drive
failed to fix things, that was _also_ discovered through the monotone
server's complaints.  If you want to paranoid, though, you can as
always run "db check".

(Hmm, it would be nice, though, if when the server finds data
corruption and freaks out, it would send an informative error to the
client, instead of just to its own log.  It's a touch tricky to do,
since it means the server has to stay up long enough for the tcp
buffers to drain, when what it would rather do is exit; there's some
special case code in their to handle this correctly for epoch
mismatch, though, that might be generalizable... any takers?)

We should especially thank Mike Shaver, who actually runs the server,
for his very responsive and helpful work in dealing with this.

We hope that things are fixed now, though I'll continue to keep an eye
on the logs...

Cheers,
-- Nathaniel

-- 
"...All of this suggests that if we wished to find a modern-day model
for British and American speech of the late eighteenth century, we could
probably do no better than Yosemite Sam."