[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Help-bash] add and expand
From: |
Val Krem |
Subject: |
Re: [Help-bash] add and expand |
Date: |
Thu, 8 Jun 2017 02:02:20 +0000 (UTC) |
Hi Greg and all,
Let me explain and give a practical example again.
I am trying to develop a pipeline procedure in SLURM
(https://hpc.nih.gov/docs/job_dependencies.html).
I have two jobs job1.sh and job2.sh. Job2.sh deepened on job1.sh and give job
dependency syntax when I am submitting job2.sh
In SLURM, when I submit the job the syntax is as follow
JJ1=$(sbatch job1.sh)
JJ1 will give the job id or PID (numeric value) for the job1.sh.
echo ${JJ1} will produce number like this 11254323
when I submit job2.sh
JJ2=$(sbatch --dependency=afterok:${JJ1} job2.sh)
In this case Job2.sh runs only if job1.sh finishes.
For single job submission it is ok.
My issue is when I am submitting in an array.
There three jobs runs separately for each city and the final job combine
all the results of the three jobs and do some analysis.
#! /bin/bash
#sbatch is a command in SLURM
city= "NY LA DC"
for tr in ${city};
do
dp[${#tr]=$(sbatch city${city}.sh );
done
When I put echo statement within the do loop
echo ${dp[${#tr}]} I got
14547
14548
14549
The next job depends on these three job ids.
if I knew these number at the time of job submission then I do rgis
comb=(sbatch --dependency=afterok:14547,14548,14549 combine.sh )
will work fine.
Since I don't know these numbers at the time of the submission,I have to pass
it as variable
Let us assume the variable newvar contain the above numbers = 14547,14548,14549
comb=(sbatch --dependency=afterok:${newvar) combine.sh )
will do the job
The complete job submission for the two jobs
#! /bin/bash
#sbatch is a command in SLURM
city= "NY LA DC"
# Job1
for tr in ${city};
do
dp[${#tr]=$(sbatch city${city}.sh );
don
#job2.
comb=(sbatch --dependency=afterok:${newvar) combine.sh )
So my question is how to get those three job ids out of the for loop.
Here is my revised attempt
#! /bin/bashcity= "NY LA DC"
count=0
# job1.
for tr in ${city};
do
dp[${#tr]=$(sbatch city${city}.sh );
array[$count]=${dp[${#tr}]}
((count++))
done
H1="address@hidden"
newvar=$(echo "$H1" | sed 's/[[:blank:]]/,/g'); export newvar
echo "$newvar"
# Job2.
comb=(sbatch --dependency=afterok:${newvar) combine.sh )
Is there a better and clean way of doing this?
On Tuesday, June 6, 2017 4:13 PM, Greg Wooledge <address@hidden> wrote:
On Tue, Jun 06, 2017 at 08:48:25PM +0000, Val Krem wrote:
> I have two jobs running one after the other (job2 run after job1). Job1 has
> three jobs running within for loop.
Background processes?
> val="Four five six"
>
> Job1.
> for a in $val
STOP THIS! Stop putting "lists" in a string variable with spaces between
things. Use an array.
> do
> Within this loop three jobs are submitted
> done
Are you running *background jobs*? Or what? What do you do inside the
loop?
> Job2. This job should run after the three jobs completed.
Are they background jobs? Did you capture their PIDs with $! one by
one? Where did you store the PIDs? Or do you simply want to call
"wait" to wait for all the background jobs to complete?
> My issue to extract the three PIDs and export them as one variable
EXPORT?! Why?
> Here is my attempt
> for tr in ${val};
I thought your loop iterator was "a", not "tr". Why did you change it?
What does "a" mean? What does "tr" mean? Why did you choose these
variables?
> do
>
> job[${#tr}]= some process;
You are creating an array named job and indexing it by the LENGTH of
the string variable tr? Huh? What?
What is "some process"?
Your assignment has a god damned SPACE after the = sign so it doesn't
even work.
> tt1="${job[${#tr}]}" #### gets the PIDs for each job
> echo $tt1
> done
Can't even bring myself to comment any more. So tired.
> The echo statement within the for loop produced the three PID like
>
> 1009
> 1010
> 1011
"... BY SOME MIRACLE I have an array with three elements."
Let's just assume you got these values somehow. They are in an array
named "job". I can't for the life of me figure out HOW you got them,
because your code isn't even code, and even your *fake* code is a
damned disaster, but let's say you have an array.
job=(1009 1010 1011)
> I want this three PIDs to be exported as one variable like,
>
> test2=$1009,$1010,$1011
I told you how to do this already. You use the [*] expansion in
single quotes to join an array into a string variable, using the
first character of IFS.
IFS=,
export test2="${job[*]}"
unset IFS
Are those dollar signs supposed to be part of the string? Then you
could do a fancypants expansion.
IFS=,
export test2="${job[*]/#/\$}"
unset IFS
WHY are you exporting this crazy string into the environment with a
useless name like "test2" and inscrutable contents? What are you doing
with it?
If you just want to launch 3 background processes and wait for them all
to finish, you don't need ANY of this crap.
for i in 1 2 3; do
./my background job "$i" &
done
wait
That's it!
- [Help-bash] add and expand, Val Krem, 2017/06/05
- Re: [Help-bash] add and expand, Peter West, 2017/06/05
- Re: [Help-bash] add and expand, Val Krem, 2017/06/05
- Re: [Help-bash] add and expand, Peter West, 2017/06/05
- Re: [Help-bash] add and expand, Greg Wooledge, 2017/06/06
- Re: [Help-bash] add and expand, Val Krem, 2017/06/06
- Re: [Help-bash] add and expand, Greg Wooledge, 2017/06/06
- Re: [Help-bash] add and expand,
Val Krem <=
- Re: [Help-bash] add and expand, Dennis Williamson, 2017/06/07
- Re: [Help-bash] add and expand, Greg Wooledge, 2017/06/08
- Re: [Help-bash] add and expand, Dennis Williamson, 2017/06/08
- Re: [Help-bash] add and expand, Greg Wooledge, 2017/06/08