[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Parallelism a la make -j <n> / GNU parallel
From: |
Mike Frysinger |
Subject: |
Re: Parallelism a la make -j <n> / GNU parallel |
Date: |
Sun, 6 May 2012 02:28:23 -0400 |
User-agent: |
KMail/1.13.7 (Linux/3.3.4; KDE/4.6.5; x86_64; ; ) |
On Saturday 05 May 2012 04:28:50 John Kearney wrote:
> Am 05.05.2012 06:35, schrieb Mike Frysinger:
> > On Friday 04 May 2012 15:25:25 John Kearney wrote:
> >> Am 04.05.2012 21:13, schrieb Mike Frysinger:
> >>> On Friday 04 May 2012 15:02:27 John Kearney wrote:
> >>>> Am 04.05.2012 20:53, schrieb Mike Frysinger:
> >>>>> On Friday 04 May 2012 13:46:32 Andreas Schwab wrote:
> >>>>>> Mike Frysinger writes:
> >>>>>>> i wish there was a way to use `wait` that didn't block until all
> >>>>>>> the pids returned. maybe a dedicated option, or a shopt to enable
> >>>>>>> this, or a new command.
> >>>>>>>
> >>>>>>> for example, if i launched 10 jobs in the background, i usually
> >>>>>>> want to wait for the first one to exit so i can queue up another
> >>>>>>> one, not wait for all of them.
> >>>>>>
> >>>>>> If you set -m you can trap on SIGCHLD while waiting.
> >>>>>
> >>>>> awesome, that's a good mitigation
> >>>>>
> >>>>> #!/bin/bash
> >>>>> set -m
> >>>>> cnt=0
> >>>>> trap ': $(( --cnt ))' SIGCHLD
> >>>>> for n in {0..20} ; do
> >>>>>
> >>>>> (
> >>>>>
> >>>>> d=$(( RANDOM % 10 ))
> >>>>> echo $n sleeping $d
> >>>>> sleep $d
> >>>>>
> >>>>> ) &
> >>>>>
> >>>>> : $(( ++cnt ))
> >>>>>
> >>>>> if [[ ${cnt} -ge 10 ]] ; then
> >>>>>
> >>>>> echo going to wait
> >>>>> wait
> >>>>>
> >>>>> fi
> >>>>>
> >>>>> done
> >>>>> trap - SIGCHLD
> >>>>> wait
> >>>>>
> >>>>> it might be a little racy (wrt checking cnt >= 10 and then doing a
> >>>>> wait), but this is good enough for some things. it does lose
> >>>>> visibility into which pids are live vs reaped, and their exit status,
> >>>>> but i more often don't care about that ...
> >>>>
> >>>> That won't work I don't think.
> >>>
> >>> seemed to work fine for me
> >>>
> >>>> I think you meant something more like this?
> >>>
> >>> no. i want to sleep the parent indefinitely and fork a child asap
> >>> (hence the `wait`), not busy wait with a one second delay. the `set
> >>> -m` + SIGCHLD interrupted the `wait` and allowed it to return.
> >>
> >> The functionality of the code doesn't need SIGCHLD, it still waits till
> >> all the 10 processes are finished before starting the next lot.
> >
> > not on my system it doesn't. maybe a difference in bash versions. as
> > soon as one process quits, the `wait` is interrupted, a new one is
> > forked, and the parent goes back to sleep until another child exits. if
> > i don't `set -m`, then i see what you describe -- the wait doesn't
> > return until all 10 children exit.
>
> Just to clarify what I see with your code, with the extra echos from me
> and less threads so its shorter.
that's not what i was getting. as soon as i saw the echo of SIGCHLD, a new
"sleeping" would get launched.
-mike
signature.asc
Description: This is a digitally signed message part.
- Re: Parallelism a la make -j <n> / GNU parallel, (continued)
- Re: Parallelism a la make -j <n> / GNU parallel, Mike Frysinger, 2012/05/04
- Re: Parallelism a la make -j <n> / GNU parallel, John Kearney, 2012/05/04
- Re: Parallelism a la make -j <n> / GNU parallel, Mike Frysinger, 2012/05/05
- Re: Parallelism a la make -j <n> / GNU parallel, Andreas Schwab, 2012/05/05
- Re: Parallelism a la make -j <n> / GNU parallel, Chet Ramey, 2012/05/07
- Re: Parallelism a la make -j <n> / GNU parallel, John Kearney, 2012/05/05
- Re: Parallelism a la make -j <n> / GNU parallel,
Mike Frysinger <=
- Re: Parallelism a la make -j <n> / GNU parallel, John Kearney, 2012/05/06
- Re: Parallelism a la make -j <n> / GNU parallel, Chet Ramey, 2012/05/04
- Re: Parallelism a la make -j <n> / GNU parallel, Mike Frysinger, 2012/05/05
- Re: Parallelism a la make -j <n> / GNU parallel, John Kearney, 2012/05/05
- Re: Parallelism a la make -j <n> / GNU parallel, Mike Frysinger, 2012/05/06
- Re: Parallelism a la make -j <n> / GNU parallel, John Kearney, 2012/05/06
- Re: Parallelism a la make -j <n> / GNU parallel, Mike Frysinger, 2012/05/06
- Re: Parallelism a la make -j <n> / GNU parallel, Chet Ramey, 2012/05/07
- Re: Parallelism a la make -j <n> / GNU parallel, Chet Ramey, 2012/05/07
- Re: Parallelism a la make -j <n> / GNU parallel, Chet Ramey, 2012/05/07