Re: Hanging with defunct processes

[Schweiss, Chip]

Something else is going on here.  The same is consistently happening without logging enabled at about 23K jobs completed.  

Any clues on how to debug the cause of this?  Is it possibly a perl problem?

On Tue, Apr 12, 2022 at 12:55 PM Schweiss, Chip <chip@innovates.com> wrote:

Seems I have exceeded the limits of GNU Parallel.  When I remove the log and resume it runs fine.   My job list is very big and so is the log:

[root@ceph-admin-prod1 work]# wc -l parallel.joblog
3701743 parallel.joblog

On Tue, Apr 12, 2022 at 10:59 AM Schweiss, Chip <chip@innovates.com> wrote:

I'm using a parallel routine for running rsyncs I have used many times before, however, when running it today for the first time on a new system all runs hang with all the children processes in a zombie state.   This is on CentOS 7.9.

Any clues as to why this would be happening?

[root@ceph-admin-prod1 sync_hcpdb]# ps -ef |grep rsync
root     196618 196498 33 10:42 pts/56   00:03:49 perl /usr/local/bin/parallel --jobs 40 --resume --joblog /ceph/hcpdb/sync_hcpdb/work/parallel.joblog --retries 3 --progress /ceph/hcpdb/sync_hcpdb/99rsync_job.sh {}
root     205121 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205122 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205132 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205142 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205146 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205151 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205164 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205177 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205216 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205290 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205309 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205330 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205337 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205340 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205345 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205349 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205355 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205362 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205370 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205380 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205392 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205401 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205410 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     205420 196618  0 10:46 pts/56   00:00:00 [99rsync_job.sh] <defunct>
root     220866 127054  0 10:53 pts/1    00:00:00 grep --color=auto rsync
[root@ceph-admin-prod1 sync_hcpdb]# /usr/local/bin/parallel --version
GNU parallel 20220322
Copyright (C) 2007-2022 Ole Tange, http://ole.tange.dk and Free Software
Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.

Web site: https://www.gnu.org/software/parallel

When using programs that use GNU Parallel to process data for publication
please cite as described in 'parallel --citation'.