cvs corruption during tag operation

[Jeremy Todd]

Hi,

I've managed to reproduce data corruption which is apparently caused by a
cvs tag operation. I'm working on finding the exact cause of the bug, but
here's what I've done thus far:

We're running CVS client 1.11.17 from a Windows XP machine, I downloaded the
binary a while ago, possible from WinCVS or Cygwin, but I'm not sure. Our
CVS server is running FreeBSD 5.4 and we connect to it via ssh (our CVSROOT
looks like :ext:x@y:/cvsroot).

We have a fairly large (7138 files, 657 MB) module containing both text and
binary files. Our build scripts check out this module and tags it fairly
often (perhaps 20-30 times each day). We noticed data corruption in a file
in this module, and I was able to reproduce it with a perl script that does
something like this:

while(true) {
  cvs co X
  cd X
  cvs tag test-n (where n is the current iteration)
  cd ..
  move X Y
  cvs co X
  diff -r X Y
}

after about 90 iterations the diff will fail. In every case, a single
character in one of the files is modified by turning ON the second bit. In
other words the hex value of the character is increased by 2.

This has happened in both text and binary files. When it happened in source
files, it resulted in a compilation error, which is how we caught the
corruption. Unfortunately it will be more difficult to determine if this has
happened in a binary file, and in those cases I fear we'll have some serious
problems down the road.

Note that when the files are corrupted, it appears to be a single byte in
the x.y,v file in the CVS repository that's corrupted. I have not yet
learned how CVS is implemented, but it seems that changes made directly to
the files in the repository can go unnoticed by CVS (no checksums?), so it's
difficult to know whether it's CVS, SSH, or the filesystem that's causing
the corruption.

Any insights on how to debug further would be most appreciated. I wasn't
able to find any similar known bugs (at least not for cvs versions as late
as 1.11.17). Unfortunately it takes about 6 hours to reproduce, so it's
going to be a challenge to say the least.

Regards,
Jeremy Todd
iZotope, Inc.