parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Replacement string for process number


From: Jay Hacker
Subject: Replacement string for process number
Date: Wed, 22 Dec 2010 09:51:21 -0500

I'd like to be able to use the number of a process in a GNU parallel command.  For instance, I want to distribute 100 files semi-evenly to 32 machines named node00 - node31.  Perhaps it would look something like this, using "{p}" as the replacement string for the process number:

$ parallel -P32 scp {} node{p}:/data ::: *.gz

That way, 3 or 4 files go to each of node00, node 01...

Or I want to concatenate 100 files semi-evenly into 16 pieces:

$ parallel -P16 "cat {} >> output-file{p}.txt" ::: ~/files/*.txt

gives 16 files named output-file00.txt ... output-file15.txt, each consisting of 6 or 7 of the input files.

It's also useful to have a replacement string for the total number of processes (the -P given, basically), maybe {P}, and the number of the input item, maybe {n}.

To clarify, it would work something like this:

$ seq 0 10 | parallel -P3 echo {n}/{p}/{P}
0/0/3 0
1/1/3 1
2/2/3 2
3/1/3 3
4/0/3 4
5/2/3 5
6/2/3 6
7/1/3 7
8/0/3 8
9/0/3 9
10/1/3 10

Though of course the order of the lines may be different.

Is there a better way to do this already, or is this something that would be straightforward to add?



reply via email to

[Prev in Thread] Current Thread [Next in Thread]