[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[bug #29512] [feature request] xargs parallel processing env-val improve

From: anonymous
Subject: [bug #29512] [feature request] xargs parallel processing env-val improvement
Date: Sat, 10 Apr 2010 15:45:44 +0000
User-agent: Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Chrome/5.0.365.0 Safari/533.3


                 Summary: [feature request] xargs parallel processing env-val
                 Project: findutils
            Submitted by: None
            Submitted on: Sat Apr 10 15:45:43 2010
                Category: xargs
                Severity: 3 - Normal
              Item Group: None
                  Status: None
                 Privacy: Public
             Assigned to: None
         Originator Name: Tomas Mudrunka
        Originator Email: address@hidden
             Open/Closed: Open
         Discussion Lock: Any
                 Release: None
           Fixed Release: None



It would be usefull to have ability to launch xargs -P with different
envinronment value for each N processes. (eg. 2 with blank, 4 with something
in this env and 4 more with something different).

Nowadays i can run some command parralely on multiple CPU cores like this:

command | xargs -n 1 -P "$cpus" some_command --

but i am thinking about processing on whole cluster of computers with sshds,
imagine that xxd command is VERY VERY CPU intensive and i will do something
like this:

0 ;) address@hidden Shared $ DPREFIX=''
0 ;) address@hidden Shared $ echo lol | $DPREFIX xxd
0000000: 6c6f 6c0a                                lol.

0 ;) address@hidden Shared $ DPREFIX='ssh harvie.cz'
0 ;) address@hidden Shared $ echo lol | $DPREFIX xxd
Host key fingerprint is 41:72:28:3d:f5:f5:d3:a2:0b:f6:e5:c7:a2:c4:b0:d3

0000000: 6c6f 6c0a                                lol.

so i can simply trigger if the process will be launched localy, or remotely.
now imagine bit more complex example:

cpus='2;4=ssh node1.harvie.cz;4=ssh node2.harvie.cz';
#now xargs will run 10 processes total.
#2 with blank DPREFIX env value
#+4 with DPREFIX='ssh node1.harvie.cz'
#+4 with DPREFIX='ssh node2.harvie.cz'
#you need to have public key copied on those nodes or something...

process() {
        cat $1 | $DPREFIX xxd > $2
        #handle failed ssh connection here
export -f process;
command | xargs -0 -n 1 -P "$cpus" bash -c 'process "$@"' --

So you will be able to run xargs on top of network cluster... Maybe this does
not fit all of possible situations, but i think sometimes it can be really

this is just IMHO most cool of them, but there are many other ways how to
make use of such feature...
eg.: you can use 4 cores, but run with bigger nice value on one of them, so
there will be some CPU power left if you will want to work on something else.


Reply to this item at:


  Message sent via/by Savannah

reply via email to

[Prev in Thread] Current Thread [Next in Thread]