Hi all, I'm trying to follow the example in the docs under the section "EXAMPLE: GNU Parallel as queue system/batch manager" but am having some issues.
I have a simple shell script that just echos a short line of data then sleeps for 1 second in a loop.
My command looks like:
tail -n+0 -f <(producer.sh) | parallel --lb -j1 --pipe --round consumer.sh
This is a representative toy example of what I'd like to do, which is have some number of long-running jobs that are fed data occasionally over time and have that data be immediately processed as it comes in.
The issue I'm seeing seems to be with batch size. If I use the default, nothing ever happens. If I set it to something small like --block 5 I get initial activity, but then the auto sizing happens if the messages happen to be larger and then processing stops.
From testing, if I manually echo data to a jobqueue file being tailed, it seems like once enough data has been written to fill blocksize, that's when the records get parsed from the block and fed to the jobs, otherwise it's just being accumulated in a buffer somewhere. Like if the current block size is 8, if I echo "hi" 4 times, after the fourth time is when all 4 lines get processed, otherwise nothing happens.
I'm on version 20160722 if that helps. Any advice is appreciated.