Re: GNU Parallel - Error:Argument list too long

[Hans Schou]

It takes a long time to sort the files and if you don't need that use -U

    ls -U | parallel do_domething

You don't have to specify amount of cpu to use. `parallel` will be greedy by default.

2017-01-16 12:49 GMT+01:00 Shlomi Fish <shlomif@shlomifish.org>:

Hi Samdani A.,

On Mon, 16 Jan 2017 16:57:47 +0530
Samdani A <samdani1593@gmail.com> wrote:

> Dear all,
>
> I used GNU parallel to process 144766 files using 8 jobs run in parallel.
> But the following error arising on starting the process.
>
> parallel -j 8 "do something" ::: *.txt
> bash: /usr/local/bin/parallel: Argument list too long
>
> Can anyone tell the maximum input files which can be given for input to run
> in GNU parallel? or anyother way to process these many number of files?
>

This is a limitation of Unix - see
https://interviews.slashdot.org/story/04/10/18/1153211/rob-pike-responds . You
can overcome it by passing the filenames using standard input, e.g:

ls | grep -vE '\.txt$' | [parallel here]


>
> Thanks and Regards
> Samdani.A



--
-----------------------------------------------------------------
Shlomi Fish       http://www.shlomifish.org/
NSA Factoids - http://www.shlomifish.org/humour/bits/facts/NSA/

Give me ASCII or give me deaþ!

Please reply to list if it's a mailing list post - http://shlom.in/reply .