[Savannah-hackers-public] stuck git-deamons on vcs.sv.gnu.org

[Assaf Gordon]

Hello,

regarding the stuck git-daemons on vcs 
(http://lists.gnu.org/archive/html/savannah-hackers-public/2016-05/msg00037.html),
this was not a one-time thing, and the stuck daemons are again accumulating.

To monitor them, I've added a small script in 
vcs:/root/agn/check-anon-vcs-daemons.sh that will show how many processes are 
there (script also attached).

===
$ ./check-anon-vcs-daemons.sh

VCS anonymous daemons (user 'nobody')

process counts:
      count stat process
          7 SN   cvs
         60 SN   git
         60 SN   git-daemon
         60 SN   git-upload-pack

10 oldest processes:

    ELAPSED  STARTED STAT   PID COMMAND
   01:11:54 14:29:58 SN   21083 cvs
   01:05:38 14:36:14 SN   27163 cvs
   01:02:39 14:39:13 SN   27503 cvs
   01:00:43 14:41:09 SN   28327 cvs
   01:00:38 14:41:14 SN   28344 cvs
      48:06 14:53:46 SN    2511 git-daemon
      48:06 14:53:46 SN    2512 git
      48:06 14:53:46 SN    2513 git-upload-pack
      48:06 14:53:46 SN    2515 git-daemon
      48:06 14:53:46 SN    2516 git
===

Trying to see why they all hang, I notice that 'git-upload-pack' seems to block 
on reading from STDIN, which is a socket:
===
vcs:~# ps -H -p 2510,2511,2512,2513
  PID TTY          TIME CMD
 2510 ?        00:00:00 timeout
 2511 ?        00:00:00   git-daemon
 2512 ?        00:00:00     git
 2513 ?        00:00:00       git-upload-pack

vcs:~# strace -p 2513
Process 2513 attached - interrupt to quit
read(0, ^C <unfinished ...>
Process 2513 detached

vcs:~# lsof -p 2513
COMMAND    PID   USER   FD   TYPE  DEVICE SIZE/OFF    NODE NAME
git-uploa 2513 nobody  cwd    DIR   253,0     4096 8813257 /srv/git/gnulib.git
git-uploa 2513 nobody  rtd    DIR   253,0     4096       2 /
git-uploa 2513 nobody  txt    REG   253,0   590364 6462611 
/usr/lib/git-core/git-upload-pack
git-uploa 2513 nobody  mem    REG   253,0 32979997 6457530 
/srv/git/gnulib.git/objects/pack/pack-63d4c0024e47875e88f77fce8ea10bafe8baad50.pack
git-uploa 2513 nobody  mem    REG   253,0  4672760 6457529 
/srv/git/gnulib.git/objects/pack/pack-63d4c0024e47875e88f77fce8ea10bafe8baad50.idx
git-uploa 2513 nobody  mem    REG   253,0  1380836 6553935 
/lib/i686/cmov/libc-2.11.3.so
git-uploa 2513 nobody  mem    REG   253,0   117367 6553949 
/lib/i686/cmov/libpthread-2.11.3.so
git-uploa 2513 nobody  mem    REG   253,0    71432 6553950 
/lib/i686/cmov/libresolv-2.11.3.so
git-uploa 2513 nobody  mem    REG   253,0    79980 6461264 
/usr/lib/libz.so.1.2.3.4
git-uploa 2513 nobody  mem    REG   253,0   118060 6553839 /lib/ld-2.11.3.so
git-uploa 2513 nobody    0u  IPv4     xxx      0t0     TCP 
vcs.savannah.gnu.org:git->XXXXXXXXXXX:55557 (ESTABLISHED)
git-uploa 2513 nobody    1u  IPv4     xxx      0t0     TCP 
vcs.savannah.gnu.org:git->XXXXXXXXXXX:55557 (ESTABLISHED)
git-uploa 2513 nobody    2w  FIFO    0,10      0t0 4745445 pipe
===

Does this ring a bell to anyone?

thanks,
 - assaf

check-anon-vcs-daemons.sh
Description: Bourne shell script