parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: very imblanced host allocation when using sshloginfile


From: Daniel LaFlamme
Subject: Re: very imblanced host allocation when using sshloginfile
Date: Sat, 11 Aug 2018 19:19:13 +0000 (UTC)

Posting this here for others benefit. I was unable to track down exactly what was causing this, but I am confident that the issue was related to the use of a remote ssh-agent for authentication. The setup was:

Host A ---[ssh]---> Host B ---[ssh]--->Hosts 01-12

I had an SSH agent running on A (in this case, Pageant) and an ssh connection from Host A to Host B that forwarded auth requests back to the agent. GNU parallel was being run from Host B. I think the agent on A got into a weird state, since when I restarted it and reestablished the connection from A to B, everything worked. I later decided to just use passphrase-less ssh keys for this and haven't seen this issue again.

On Friday, July 13, 2018, 2:19:31 AM EDT, Ole Tange <ole@tange.dk> wrote:


On Wed, Jul 11, 2018 at 3:45 AM, Daniel LaFlamme

<misterlaflamme@yahoo.com> wrote:
> Hit send prematurely.
>
> Later in the job run, I did start seeing messages like the following:
>
> "mux_client_request_session: session request failed: Session open refused by
> peer"
>
> All machines had the same low load during parallel's run. I have spot
> checked sshd_config on a couple of the servers and they are the same. They
> are also managed by puppet so they should all have the same configuration.
>
> My question is: what can I do to make what is going on under the hood more
> transparent so I can debug the problem further?


-t will show what GNU Parallel actually runs the moment it is being run.

So you will be getting lines like:

ssh -S /tmp/control_path_dir-dTad/ssh-%r@%h:%p server -- exec perl -e
@GNU_Parallel\\\=split/_/,\\\"use_IPC::Open3\\\;_use_MIME::Base64\\\"\\\;eval\\\"@GNU_Parallel\\\"\\\;\\\$chld\\\=\\\$SIG\\\{CHLD\\\}\\\;\\\$SIG\\\{CHLD\\\}\\\=\\\"IGNORE\\\"\\\;my\\\$zip\\\=\\\(grep\\\{-x\\\$_\\\}\\\"/usr/local/bin/bzip2\\\"\\\)\\\[0\\\]\\\|\\\|\\\"bzip2\\\"\\\;open3\\\(\\\$in,\\\$out,\\\"\\\>\\\&STDERR\\\",\\\$zip,\\\"-dc\\\"\\\)\\\;if\\\(my\\\$perlpid\\\=fork\\\)\\\{close\\\$in\\\;\\\$eval\\\=join\\\"\\\",\\\<\\\$out\\\>\\\;close\\\$out\\\;\\\}else\\\{close\\\$out\\\;print\\\$in\\\(decode_base64\\\(join\\\"\\\",@ARGV\\\)\\\)\\\;close\\\$in\\\;exit\\\;\\\}wait\\\;\\\$SIG\\\{CHLD\\\}\\\=\\\$chld\\\;eval\\\$eval\\\;
QlpoOTFBWSZTWVeMEdsAABWfgHV////u538ev//f/jABa1BsRJonknlDIaaNqNNA9RoAABoDQANCKnkwmmmkaABo0ABoNAAADIAJRETyIZTEwJk0aGmhkDQGIyGhobREc4dRYSEyZfhTy06+rZhU0MqRGZKf8Ax8DK579pAO0n455KoEgINRBiGuhRpZlhO3wPCiBlr6zK6GXeeLInJeWvFYUY4ISvgw6W5WsaU2Wr1q6QFB4rRqOcWIJy707Ay5C7FiIrLuy/IkSVHTPaON8Rm8dc4hA4KINl8gOkchptIKBW6VBgMJRomCqAtu2esksMlwL6BeDvOXJlyqESHrA8VmTNG0e8PAMyDM4xDRKSGnQQvEbBzUmk+Q9kI72hZQbNMcWvPqJCb2z32LrFGMRU4XjkZfhHdHa9V2DoDnTDMUYBsy77issbk6YjbTiCxK8gCTG/H+2oSpqb5rSr1kWpSlGtHmiVIJrZCqSpbGJT200f8MKHBMpRjrrcaSVVnmx24XapGhXnjCIHmAekOqOTFwIJY7QOBdssV1wGMYm2NoNJUbuLCMzUBJ34szrbLEUC+yxP4u5IpwoSCvGCO2;

Given that the problem looks like being related to ssh, you can try
running some:

  ssh -S /tmp/control_path_dir-dTad/ssh-%r@%h:%p server sleep 10

by hand.

Since there _are_ jobs on all your hosts, then I reckon it is a
problem in your local setup. If find a solution, please post it. It
may be that others have the same issue.


/Ole


reply via email to

[Prev in Thread] Current Thread [Next in Thread]