Re: Serious cvs tag/rtag issues Please help

[Todd Denniston]

Pranab Panda (pkpanda) wrote, On 10/05/2007 10:39 AM:
> Thanks a lot Todd
>
> As Per IT the server used to go to Hang state and they
> Had to reboot to get it back alive.
> The problem started happeninig after 
> we moved cvs repository filesystem from
> SAN(attached storage) to NAS(filer)
> IT folks say that we are hitting a known NFS bug
> With this old version of Red HAT.
>
> Yesterday I had to clear our all locks left over by the
> Crash and then I got errors for tag/rtag.
> I panicked seeing that error message, but 
> LUCKILY when I tried the tag/rtag second time
> It worked!

IIRC the first one warned you there had been a problem, but cleared the lock, 
so the second could work. That was why I was suggesting looking for the lock 
files first so you could find appropriate users and have them finish their 
checkins... to try and get the baseline to a coherent point.

> We are moving back to SAN to avoid the crash,
> But long term will upgrade OS on the cvs server.

three suggestions till then:
0) make a backup yourself on off-line media before doing much else.

1) go ahead and run one of check_cvs || validate_repo on your repository, 
because you may have other (worse) problems created by having your repo on an 
NFS mounted drive... the list has seen it in the past, and the sooner you know 
about a problem the more likely it is you can recover from it.  The named 
script(s) do a reasonable job finding the ones we have seen.

2) check to see if they are hard mounting or soft mounting the NFS repo on the 
CVS server machine...
_MY_ experience has been that soft mounted NFS structures are very prone to 
file destruction faults, i.e., data loss.
Oh, I am making the assumption that you are using one of :ext: :pserver: from 
other machines to connect to the CVS server machine and not accessing the NFS 
share directly from any other machine. :}

> Best regards
> pranab
>
> -----Original Message-----
> From: Todd Denniston [mailto:address@hidden 
> Sent: Friday, October 05, 2007 7:19 AM
> To: Pranab Panda (pkpanda)
> Cc: info-cvs list
> Subject: Re: Serious cvs tag/rtag issues Please help
>
> Pranab Panda (pkpanda) wrote, On 10/05/2007 01:17 AM:
> >  
> > Hi All
> >
> > In last couple of days our cvs server has gone done few times,
>
> You mean like power outage/hardware fail???
> UGH.
>
> > Checkin and checkout work fine, but
> > since this afternoon I see errors with cvs tag and rtag commands.
> > They both fail with following error
> > cvs [rtag aborted]: could not open lock file
> > `/auto/cvsroot/repository/Files/,file12,': File exists
> >
> > I am clueless how to fix this.
> > Please help if know whats the issue
> > this is very critical!
> >
> > NOTE: cvs --version
> > Concurrent Versions System (CVS) 1.11.2 (client/server) On Linux 
> > 2.4.21-27.0.1.ELsmp #1 SMP Mon Dec 20 18:47:45 EST 2004 i686
> > i686 i386 GNU/Linux
> >
> > Thanks in advance
> > pranab
>
> Suggestions:
> 1) make a backup[1] of what you have... immediately.
> 2) if it was power/hardware fail, A) you should get new UPS/hardware, B)
> probably fsck the drive with the repo on it.
> 3) `locate check_cvs; locate validate_repo` one of these should be on
> the system in a cvs/contrib directory (assuming a nice Linux distro),
> Otherwise you can beat the .in file[2] into shape.
> 4) `perldoc check_cvs || perldoc validate_repo`
> 5) use one of check_cvs || validate_repo on your repository to see what
> other problems you have.
>
> 6) with the ",filename," files, I _think_ you should see if there are
> usernames on them (if you are using pserver there might not be real
> ones) and if so ask those users to finish their broken commits.
> 7) if you think it appropriate, make a new backup.
> 8) ask the group again about help with the new problems if needed.
> 9) upgrade to a much newer CVS/verify RH has updated for all the
> security problems found since 1.11.2.
>
> Good luck.
>
> [1] http://ximbiot.com/cvs/manual/cvs-1.11.22/cvs_2.html#SEC24
> [2]
> http://cvs.savannah.nongnu.org/viewvc/*checkout*/ccvs/contrib/validate_r
> epo.pl?root=cvs&content-type=text%2Fplain


--
Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane)
Harnessing the Power of Technology for the Warfighter
http://www.crane.navy.mil/custfeedback