Re: Running >10 jobs in parallel on the same remote machine

[Thomas Sattler]

> I would like to have people test the idea before I implement
> it. I need your help with the options:
> 
>   ssh <<what options here?>> server sleep 10000000 &
>   sleep 3 # To make sure the above is logged in and set up
>   seq 1000 | parallel -j0 ssh <<what options here?>> server 'echo {};sleep 2'
>   kill %1 # Kill the sleeping ssh

According to the ssh_config manual, the extra ssh needs the
options ControlMaster and ControlPath, the others only need
ControlPath (as ControlMaster defaults to "no", which means
check for existing 'masters' but don't try to become one).

It can be tested like this:

 1) in one terminal run:

    ssh -o ControlMaster=yes -o ControlPath=/tmp/foo \
      server "date; read; last | head"

 2) as soon as you can see the output of 'date' you can
    run the following in another terminal:

    seq 100 | parallel -j0 ssh -o ControlPath=/tmp/foo \
      server 'echo {};sleep 2'

 3) Now press [Enter] in the first terminal and see the
    output of 'last': There was only one login.

ControlPath should be a "little bit better" than the one
above, something like "~/.parallel/%l -> %r@%h:%p" (see
man ssh_config for their meaning) but that would need
quoting that I want to avoid for the testing phase.
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

The above works fine on my Gentoo machine. Afterwards I
tested on Debian Squeeze and it just doesn't work. :-(

Even so I'm still unhappy with the idea of "one separate
ssh login per job". How about letting parallel doing the
multiplexing on its own, running a shell or perl on the
remote host to spawn the jobs?

Thomas