Re: Properly setting $Global::max_file_length to avoid line 12170 error

[Nagle, Michael F]

Correct. I have a few lines of bash code to automatically prepare the jobs file I just sent. Immediately after that file is made, it is passed as the $job_list_name variable to...

parallel --results output -a $job_list_name

Please let me know if any other information would be helpful, and thanks for all the suggestions and thought into this...

I've also attached the notebook with all relevant code. I've confirmed the issue happens whether in Jupyter or the command line.

Michael Nagle

Michael Nagle

PhD Candidate, Molecular and Cellular Biology

Forest Biotechnology Laboratory

Oregon State University

301-974-7221 (cell)

On October 5, 2022 at 5:32 GMT, Rob Sargent <robjsargent@gmail.com> wrote:

[This email originated from outside of OSU. Use caution with links and attachments.]

On 10/4/22 23:12, Nagle, Michael F wrote:

No, I'm not using backslash continuation lines. I ctrl-F'd for backslashes and see none in my jobs file passed to parallel -a

Please find attached that jobs file. Perhaps it could provide some clue. However, the same format and script used to prepare it worked in Ubuntu 20.04 LTS and it's only now that I'm using 22.04 LTS that I'm having this problem.

That's quite interesting.  I see no unreasonably long paths in the job file you sent.  Some step in the process might be quoting the entire line (or at least the arg list).

I take it your "$job_list_name" might be the name of the file you sent?

Michael Nagle

Michael Nagle

PhD Candidate, Molecular and Cellular Biology

Forest Biotechnology Laboratory

Oregon State University

301-974-7221 (cell)

On October 5, 2022 at 4:44 GMT, Rob Sargent <robjsargent@gmail.com> wrote:

[This email originated from outside of OSU. Use caution with links and attachments.]

On 10/4/22 22:33, Nagle, Michael F wrote:

Rob Sargent

[This email originated from outside of OSU. Use caution with links and attachments.]

The backslashes appear to be coming from parallel. My command has file paths with forward slashes and these are replaced with \_ by parallel, it seems.

To demonstrate, here is one my first command in the a I pass to parallel:
gemma -bfile ./1323_cohort_maf01_geno10.snp.pass -p ./pheno_files/callus_regeneration_PC1-uv_unique_rmoutliers_boxcox_resid.noheader.pheno -k ./1323_cohort_maf01_geno10.cXX.txt -lmm 1 -n 3 -miss 0.1 -o callus_regeneration_PC1-uv_unique_rmoutliers_boxcox_resid-ResidOverPhase

Here is the error for that command:
mkdir output/1/gemma -bfile .\_1323_cohort_maf01_geno10.snp.pass -p .\_pheno_files\_callus_regeneration_PC1-uv_unique_rmoutliers_boxcox_resid.noheader.pheno -k .\_1323_cohort_maf01_geno10.cXX.txt -lmm 1 -n 3 -miss 0.1 -o callus_regeneration_PC1-uv_unique_rmoutliers_boxc: Invalid argument at /home/gmobot/anaconda3/envs/gemma/bin/parallel line 12170.

Are you using backslash continuation lines by any chance?

_

Run GEMMA in parallel using SNPs filtered at MAF 0.01 and regression of traits over phase, as done previously for regeneration GWAS¶

We will use the GNU parallel framework to multithread GEMMA over phenotypes.
Tange, O. (2020, November 22). GNU Parallel 20201122 ('Biden'). Zenodo. https://doi.org/10.5281/zenodo.4284075

In this notebook, we use traits that have been processed by regression over phase indicator variable (all 69 phases) and collection of residuals. This will allow us to include the phase information in our models without dealing with a massive covarite matrix or combining phases.

In this version of the notebook, I'm troubleshooting an issue involving an error with parallel.

In [1]:

pwd

/media/gmobot/data/NSF_GWAS/notebooks/Transformation_GWAS/03_GEMMA

In [2]:

job_list_name=GEMMA_transformation_a1.jobs

If job file already exists, we want to overwrite it.

In [3]:

rm -rf $job_list_name

change $outname to test

Prepare jobs file¶

Whether or not we use backslashes in the code used to produce the jobs file, the file itself does not contain backslashes, and we encounter the same error.

No backslahes to prepare jobs file¶

In [4]:

for file in `ls ./pheno_files | grep -i "noheader"`
do
 outname=$(echo $file | sed -e "s/\\.noheader\\.pheno//g")-ResidOverPhase
 outname=$(basename $outname)
 echo "gemma -bfile ./1323_cohort_maf01_geno10.snp.pass -p ./pheno_files/$file -k ./1323_cohort_maf01_geno10.cXX.txt -lmm 1 -n 3 -miss 0.1 -o $outname" >> $job_list_name
done

Version with backslashes to prepare jobs file¶

In [5]:

# for file in `ls ./pheno_files | grep -i "noheader"`
# do
#  wc -l ./pheno_files/$file
#  outname=$(echo $file | sed -e "s/\\.noheader\\.pheno//g")-ResidOverPhase
#  outname=$(basename $outname)
#  echo "gemma -bfile ./1323_cohort_maf01_geno10.snp.pass \
# -p ./pheno_files/$file \
# -k ./1323_cohort_maf01_geno10.cXX.txt \
# -lmm 1 \
# -n 3 \
# -miss 0.1 \
# -o $outname" >> $job_list_name
# done

Inspect the jobs file¶

In [6]:

wc -l $job_list_name

53 GEMMA_transformation_a1.jobs

In [7]:

head -n 2 $job_list_name

gemma -bfile ./1323_cohort_maf01_geno10.snp.pass -p ./pheno_files/callus_regeneration_PC1-uv_unique_rmoutliers_boxcox_resid.noheader.pheno -k ./1323_cohort_maf01_geno10.cXX.txt -lmm 1 -n 3 -miss 0.1 -o callus_regeneration_PC1-uv_unique_rmoutliers_boxcox_resid-ResidOverPhase
gemma -bfile ./1323_cohort_maf01_geno10.snp.pass -p ./pheno_files/callus_regeneration_PC2-uv_unique_rmoutliers_boxcox_resid.noheader.pheno -k ./1323_cohort_maf01_geno10.cXX.txt -lmm 1 -n 3 -miss 0.1 -o callus_regeneration_PC2-uv_unique_rmoutliers_boxcox_resid-ResidOverPhase

set max_file_length globalenv¶

We encounter the same issue whether max_file_length is set to 40 or 4000

In [8]:

#export max_file_length=40

In [9]:

export max_file_length=4000

In [10]:

echo $max_file_length

4000004l

Deploy parallel over GEMMA¶

In [11]:

conda activate gemma

(gemma) 

In [12]:

parallel --results output -a $job_list_name

mkdir output/1/gemma -bfile .\_1323_cohort_maf01_geno10.snp.pass -p .\_pheno_files\_callus_regeneration_PC1-uv_unique_rmoutliers_boxcox_resid.noheader.pheno -k .\_1323_cohort_maf01_geno10.cXX.txt -lmm 1 -n 3 -miss 0.1 -o callus_regeneration_PC1-uv_unique_rmoutliers_boxc: Invalid argument at /home/gmobot/anaconda3/envs/gemma/bin/parallel line 12170.
(gemma) 

In [ ]: