RE: gnu parallel supresses output of ssh and scp when using --line-buffer option

[Warner, Ben]

Sorry, I left off the “parallel –version” command from my previous email.

 

myuser@myhost:~ % parallel --version

GNU parallel 20140422

Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014 Ole Tange and Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

GNU parallel comes with no warranty.

 

Web site: http://www.gnu.org/software/parallel

 

When using programs that use GNU Parallel to process data for publication please cite:

 

O. Tange (2011): GNU Parallel - The Command-Line Power Tool,

;login: The USENIX Magazine, February 2011:42-47.

 

 

From: Warner, Ben
Sent: 30 March 2015 14:05
To: 'parallel@gnu.org'
Subject: gnu parallel supresses output of ssh and scp when using --line-buffer option

 

Hi Ole,

At your suggestion, am moving this issue to the parallel mailing list.

 

Original issue can be found here: http://stackoverflow.com/questions/29281101/gnu-parallel-supresses-output-of-ssh-and-scp-when-using-line-buffer-option

 

Original issue:

I am trying to use gnu parallel to run some ssh and scp commands.

When I do NOT use -u or --line-buffer options, I can see the output for each thread after the thread has finished.

Because my ssh commands take a while I want to see output as it occurs.

When I try to use -u tagging does not work.

When I try to use --line-buffer, I don't get any of the output, even after threads have finished.

I've written this simple test command which behaves in the same way:

parallel -q ssh user@{} 'ls && sleep 4 && ls' ::: host1 host2

I have been looking at this very similar question http://unix.stackexchange.com/questions/101360/scp-does-not-display-output-when-used-with-gnu-parallel but the suggested solution of using the script command did not work for me. Might be because my system is not linux. (Am using Solaris).

Any help would be appreciated.

Ole’s suggested debugging:

 

parallel --version

parallel --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c

parallel --tag --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c

parallel -S host1 --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c

parallel -S host2 --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c

parallel -S host1,host2 --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c

parallel -S host1 --tag --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c

parallel -S host2 --tag --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c

parallel -S host1,host2 --tag --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c

 

This was my output:

 

myuser@myhost:~ % parallel --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c

myuser@myhost:~ % parallel --tag --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c

myuser@myhost:~ % parallel -S host2 --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c
bash: parallel: command not found
parallel: Warning: Could not figure out number of cpus on host2 (). Using 1.

myuser@myhost:~ % parallel -S host1 --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c
bash: /app/axle/bin/parallel: Permission denied
parallel: Warning: Could not figure out number of cpus on host1 (). Using 1.

myuser@myhost:~ % parallel -S host2,host1 --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c
bash: parallel: command not found
parallel: Warning: Could not figure out number of cpus on host2 (). Using 1.
bash: /app/axle/bin/parallel: Permission denied
parallel: Warning: Could not figure out number of cpus on host1 (). Using 1.

myuser@myhost:~ % parallel -S host2 --tag --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c
bash: parallel: command not found
parallel: Warning: Could not figure out number of cpus on host2 (). Using 1.

myuser@myhost:~ % parallel -S host1 --tag --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c
bash: /app/axle/bin/parallel: Permission denied
parallel: Warning: Could not figure out number of cpus on host1 (). Using 1.

myuser@myhost:~ % parallel -S host2,host1 --tag --line-buffer 'echo {}; sleep 4; echo {}' ::: a b c
bash: parallel: command not found
parallel: Warning: Could not figure out number of cpus on host2 (). Using 1.
bash: /app/axle/bin/parallel: Permission denied
parallel: Warning: Could not figure out number of cpus on host1 (). Using 1.

 

 

It is indeed interesting that even the first command (with no remoting) behaves in the same way!  i.e. no output.

 

Running with the –u option shows expected output:

 

myuser@myhost:~ % parallel -u 'echo {}; sleep 4; echo {}' ::: a b c

a

b

c

a

b

c

 

 

And running with no flag shows expected output as well:

 

myuser@myhost:~ % parallel 'echo {}; sleep 4; echo {}' ::: a b c

c

c

b

b

a

a

 

Any help would be greatly appreciated.

 

Cheers,

Ben

---

This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ Bank New Zealand Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication.