Re: Transfer output of a command from remote computer

[Ole Tange]

On Fri, May 9, 2014 at 11:44 AM, V A Ramesh <varavind121@yahoo.com> wrote:

> I am running a job like this
>
> parallel --eta -S .. --trc {.}.out < Testing_Para
>
> Testing_para:
:
> svm-train -c 0.125 -g 0.015625 -v 10 -m 2000 Combined_Full.scaled >&
> Combined_Full.scaled_0.125_0.015625.out
> svm-train -c 0.125 -g 0.03125 -v 10 -m 2000 Combined_Full.scaled >&
> Combined_Full.scaled_0.125_0.03125.out
:
> svm-train -c 0.25 -g 0.00048828125 -v 10 -m 2000 Combined_Full.scaled >&
> Combined_Full.scaled_0.25_0.00048828125.out
> svm-train -c 0.25 -g 0.0009765625 -v 10 -m 2000 Combined_Full.scaled >&
> Combined_Full.scaled_0.25_0.0009765625.out

> Problem 1:
>
> I want '.out' files after completion of the job. I realized that --trc
> {.}.out is not the appropriate way to use and I manually retrieved the
> results back from the remote computer.

When you pass the whole command as argument {.} will contain the whole
command. That is clearly not what you want.

It seems you want to run 'svm-train' on different values and want an
output file to depend on this. To simplify your command let us assume
you want the output from:

    svm-train a 1
    svm-train a 2
    svm-train a 3
    svm-train a 4
    svm-train a 5
    svm-train a 6
    svm-train a 7
    svm-train b 1
    svm-train b 2
    svm-train b 3

You can the generate the commands using GNU Parallel:

    parallel --xapply svm-train {1} {2} ::: a a a a a a a b b b ::: 1
2 3 4 5 6 7 1 2 3

If you would rather have the 'a a a a a a a b b b'  and '1 2 3 4 5 6 7
1 2 3' in files, use :::: possibly with --colsep.

Now you want the output put in files depending on the arguments:

    parallel --xapply svm-train {1} {2} ">&" out.{1}.{2} ::: a a a a a
a a b b b ::: 1 2 3 4 5 6 7 1 2 3

To return these back and remove them from the remote site you need
--return --cleanup:

    parallel --return out.{1}.{2} --cleanup --xapply svm-train {1} {2}
">&" out.{1}.{2} ::: a a a a a a a b b b ::: 1 2 3 4 5 6 7 1 2 3

But if you are simply redirecting output into a file, why not use
--results which is made for this kind of problem:

    parallel --results outdir --xapply svm-train {1} {2} ::: a a a a a
a a b b b ::: 1 2 3 4 5 6 7 1 2 3 >/dev/null

Now the output is neatly stored in a structured way in 'outdir/*/*/*/*'

> Problem 2:
>
> I also had to transfer input files from host to remote computer manually.
> How to solve this two problems.

If the argument is the inputfile, then you use --transfer. I do not
see an inputfile in your example.

If the inputfile is more like a read-only database that does not
change, but remains the same for every job, use --basefile.

> Vakkalagadda A Ramesh
> Jr. Research Fellow
> Lab of Computational Biology
> Centre for DNA Fingerprinting and Diagnostics (CDFD)

I am hoping your research results in published articles.


/Ole