Re: parallel on Amazon AWS after logoff

[Ole Tange]

On Sun, Mar 8, 2015 at 9:12 PM, andreas wpv <andreas.wpv@gmail.com> wrote:

> I am trying to scan (our) website with a script, that's basically a more
> detail wget command, that takes $1 as the url.
>
> nohup parallel -eta -j20 --line-buffer '.  extendedwget.sh  {}  >
> $RANDOMoutfile ' :::: www.test.com  > /dev/null 2>&1 &
>
> Somehow the redirection to the outfile (with $RANDOMoutfile) does not work.

wget prints to STDERR so what you are doing is sending it to
/dev/null. --eta prints to STDERR and is thus also ignored by you.

Also you ask GNU Parallel to read a file called www.test.com: I would
think that you want to that to be treated as an argument. GNU Parallel
tells you the file does not exist, but you ignore the output by
redirecting STDERR to /dev/null.

> I had it in the script previously, but then I get a lot of different
> outnames.  I also tried 4x : and 3x :, that seemed not to make a difference.

Because you are sending STDERR to /dev/null

Please do not redirect STDERR before you have a working solution. It
is like ripping out your eyes: you will not be able to see the errors,
but that does not mean they are not still there.

If you are only running this on a single argument (www.test.com) it
makes very little sense to use GNU Parallel: GNU Parallel does not go
into a command and parallelizes the internals, instead it runs
multiple commands in parallel - typically the same command but with
different arguments. If this is unclear to you, go through the
tutorial: man parallel_tutorial

> I used this with another script, and it seemed to work fine (except that it
> ran only on one core )
>
> nohup parallel [ left out details ] > /dev/null 2>&1 &
>
> Do you have a tip how to fix this?

Start by making it possible to get error messages. Sending them to
/dev/null will not help you.

My guess is you want something like:

    parallel -j20 --line-buffer --results outdir bash extendedwget.sh
{} ::: www.test.com more.websites.here

This way you get output on the screen and a copy in outdir.

nohup is really bad for debugging. Only use nohup when you are 100%
sure what you are doing. Instead you might want to use tmux:

    tmux new-session -d "parallel -j20 --line-buffer --results outdir
bash extendedwget.sh  {} ::: www.test.com more.websites.here"

This way you can logout (just like nohup), but you can later log back
in and attach to tmux and see what is happening:

    tmux attach


/Ole