Re: Dynamically changing remote servers list

[Ole Tange]

On Wed, Aug 13, 2014 at 12:17 AM, Douglas A. Augusto
<daaugusto@gmail.com> wrote:

> Firstly, thank you all for contributing to the development of GNU Parallel, it
> is a very useful tool.

Thanks. If you like GNU Parallel:

* (Re-)walk through the tutorial if you have not done so in the past year
(http://www.gnu.org/software/parallel/parallel_tutorial.html)
* Give a demo at your local user group/team/colleagues
* Post the intro videos and tutorial on Reddit/Diaspora*/forums/blogs/
Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
* Request or write a review for your favourite blog or magazine
* Invite me for your next conference

If you use GNU Parallel for research:

* Please ask your researchers to cite GNU Parallel in their
publications (use --bibtex)

If GNU Parallel saves you money:

* (Have your company) donate to FSF https://my.fsf.org/donate/

> Suppose the following scenario. There are lots of parallel jobs running 
> locally
> and remotely, but in the meantime additional remote servers become available.
> It would be nice if those new servers could be included to the pool of compute
> servers without restarting the current execution.
>
> So, instead of statically defining the servers (via sshlogin or sshloginfile),
> I am wondering if it is possible to include/exclude servers on-the-fly. This
> feature would be analogous to the way in which the load can be dynamically
> configured by simply changing a file ("procfile").

--sshloginfile already takes a file, so it will be natural to re-read
that. Probably using this method:

    When a job finishes, and it is more than 1 second since we checked last:
      Check if the file has changed modification time. If yes: re-read it.

It ought to be fairly easy to _add_ new servers. It will probably
screw up the timings in --eta/--bar, but that is hardly a serious
problem.

Removing is, however, a completely different ballgame: What do you do
about jobs currently running on the servers? Also there is no
infrastructure to say: Don't start new jobs on this server and remove
it when the last job completes. The easiest is probably to add a 'dont
start new jobs' flag to the server-object, and leave the data
structure in place. It will, however, cost a few cycles to skip the
server every time a new job is started.

--filter-hosts does the "removal" by not adding the host in the first place.

Now added to the wishlist https://savannah.gnu.org/bugs/index.php?42983


/Ole