Re: Parallel Enhancement request.

[Ole Tange]

On Wed, Jan 12, 2011 at 7:59 PM, Mike Rochford <mike@mikerochford.com> wrote:

> I have a  need to run a single command or script on many machines in a very
> large environment (>10k hosts.) Please consider adding this option to the
> parallel command. Below is an example of running a simple command.

I have considered making something similar, but refrained from it
because there are so many competitors (dssh, pssh, clusterssh, cssh,
dsh, sshpt, pussh, pdsh - and there are probably more)

I may revise my stand if you can provide me with a review of all the
tools you have tested and of the tools listed above and why they fail
to meet your requirements. Please write the review so it can be
included as sections below:
http://www.gnu.org/software/parallel/man.html#differences_between_gnu_parallel_and_alternatives

>
> cat hostlist | parallel -j 25 "ssh -l root {} uptime"
>
> Options we are looking for in the addition:
>
> - New option to execute a command/script on list of hosts provided via
> filename or piped into the command

My idea is to use --sshloginfile and make an option called
--run-on-all-hosts. Each command that GNU Parallel would normally run
would then be run on all hosts given in --sshlogin or --sshloginfile.
If GNU Parallel would normally run several commands they would all be
run on all hosts in parallel.

> - Option to specify how many hosts command/script should be sent to at a
> time {i.e. only run on 25 hosts at a time for 1000 servers}

-j would do that

> - Option to timeout the command after nnn seconds {i.e. Host is not
> responding}

I probably cannot do the host not responding, but I can do a --timeout
so if login + executing the command takes longer, it will be killed.
--timeout could even be generally useful.

> - Option to specify username to use on target host

My idea is to use --sshloginfile. This is where you will put your
hostlist and login name.

> - Option to specify a “options” file – This will have one option per line

Explain why -J is not good enough.

> - Display status optionally while running

Explain why --progress/--eta is not good enough.

> - Output related:
>
>    - the output from command/script should all accumulated in a single file

Is there any reason why you cannot just redirect it yourself: > outputfile

>    - Each line of output should have “hostname:” prefixed to all the lines

I am not too fond of messing with the output. It is one of the
fundamental ideas of GNU Parallel not to mess with the output but pass
it straight on.

But I will give you an environment variable $PARALLEL_HOST or
$SSHLOGIN that you can use. You can then: | sed /^/$PARALLEL_HOST/

>    - Output should be sorted by hostname

If you do the prefixing above you can simply: | sort

>    - Hosts that didn’t respond should be provided in a separate file

Would printing a warning message to STDERR be OK?


/Ole