I seem unable to fix my space in filenames issue using switches like -print0 for gnu-find and -0 for gnu-parallel, gnu-xargs in this scenario as is usually recommended.
I succeeded in combining find, parallel in pipe mode and xargs to run commands in parallel in "blocks" for 100k+ files. I use echo and ls in the examples below but I plan to use my own python command. Note that I want to run each command instance on more than one file due to overhead in starting my program hence the use of parallel in --pipe mode and --block etc. The command
find ./dirNames/ -type f | parallel --pipe --block 100 -j4 --round-robin \
"echo \"Start *****\"; cat ; echo \"Done *****\""
results in
Start *****
./dirNames/bbbbbbbbbbbbbbbb
./dirNames/dddddddddddddddddddd
./dirNames/aaaaaaaaaaaaaaaa
Done *****
Start *****
./dirNames/cccccccc cccccccc
./dirNames/eeeeeeeeeeeeeeeeeeee
Done *****
as desired. gnu-echo is run twice, in one instance it is run with 3 files and in the other instance with 2 files. If I try this with xargs and ls I run into the classic space in filename problem ...
find dirNames/ -type f | parallel --pipe --block 40 -j4 --round-robin \
"echo \"Start *****\"; xargs ls -l ; echo \"Done *****\""
Resulting in this
Start *****
-rw-rw-r-- 1 robert robert 0 Jun 24 10:10 dirNames/bbbbbbbbbbbbbbbb
-rw-rw-r-- 1 robert robert 0 Jun 25 16:11 dirNames/eeeeeeeeeeeeeeeeeeee
Done *****
Start *****
-rw-rw-r-- 1 robert robert 0 Jun 24 10:10 dirNames/aaaaaaaaaaaaaaaa
Done *****
Start *****
-rw-rw-r-- 1 robert robert 0 Jun 25 16:11 dirNames/dddddddddddddddddddd
Done *****
Start *****
Done *****
ls: cannot access 'dirNames/cccccccc': No such file or directory
ls: cannot access 'cccccccc': No such file or directory
which in this scenario I seem unable to fix using switches like -print0 for find and -0 for parallel and xargs as is usually recommended for this problem. parallel seems confused by the output of find with -print0. Please advise as I have truly run out of ideas :(