Re: Properly setting $Global::max_file_length to avoid line 12170 error

The backslashes appear to be coming from parallel. My command has file paths with forward slashes and these are replaced with \_ by parallel, it seems.

To demonstrate, here is one my first command in the a I pass to parallel:
gemma -bfile ./1323_cohort_maf01_geno10.snp.pass -p ./pheno_files/callus_regeneration_PC1-uv_unique_rmoutliers_boxcox_resid.noheader.pheno -k ./1323_cohort_maf01_geno10.cXX.txt -lmm 1 -n 3 -miss 0.1 -o callus_regeneration_PC1-uv_unique_rmoutliers_boxcox_resid-ResidOverPhase

Here is the error for that command:
mkdir output/1/gemma -bfile .\_1323_cohort_maf01_geno10.snp.pass -p .\_pheno_files\_callus_regeneration_PC1-uv_unique_rmoutliers_boxcox_resid.noheader.pheno -k .\_1323_cohort_maf01_geno10.cXX.txt -lmm 1 -n 3 -miss 0.1 -o callus_regeneration_PC1-uv_unique_rmoutliers_boxc: Invalid argument at /home/gmobot/anaconda3/envs/gemma/bin/parallel line 12170.

On October 5, 2022 at 4:27 GMT, Rob Sargent <robjsargent@gmail.com> wrote:

[This email originated from outside of OSU. Use caution with links and attachments.]
On 10/4/22 21:41, Nagle, Michael F wrote:
Hello,

Thanks in advance to the developers and community. I'm a biologist who is not an expert in Linux and am dealing with a pretty rudimentary issue I haven't been able to solve.

In short, I need to figure out the correct way to change $Global::max_file_length to avoid an error resulting from output folder path name being too long, or to figure out a simple workaround.

When I launch `parallel` to run over some jobs...
parallel --results output -a $job_list_name

I get:
mkdir output/1/gemma -bfile .\_1323_cohort_maf01_geno10.snp.pass -p .\_pheno_files\_aaaaatest.noheader.pheno -k .\_1323_cohort_maf01_geno10.cXX.txt -lmm 1 -n 3 -miss 0.1 -o testa1: Invalid argument at /home/gmobot/anaconda3/envs/gemma/bin/parallel line 12170.

Looking at line 12170, I see:
12163             # mkpath failed: Argument probably too long.
12164             # Set $Global::max_file_length, which will keep the individual
12165             # dir names shorter than the max length
I have tried changing max_file_length to all sorts of values, from 40 to 4000, by the two methods:
1. max_file_length=40
2. export max_file_length=4000

I then echo'd $max_file_length to make sure it was set as intended before running parallel again. I'm still getting the same issue.

This issue is happening with both the Biden and Elizabeth releases of GNU Parallel, installed via conda. It did not happen when running the same code on the same computer with Ubuntu 20.04 LTS, but is now happening with Ubuntu 22.04 LTS.

What am I doing wrong? What's the correct way to set the max_file_length global variable, and what should I set it to so I can avoid this issue?

As an alternative solution, is there a simple way for me to run GNU parallel that doesn't create a folder based on the names of my (very long) commands? I suppose I could produce a separate wrapper script for each command and pass that, but would prefer a more elegant and less tedious solution.

Thanks!
Your "I get:" bit is using backslashes. Is that from you or the parallel?

From:	Nagle, Michael F
Subject:	Re: Properly setting $Global::max_file_length to avoid line 12170 error
Date:	Wed, 5 Oct 2022 04:33:37 +0000