List of background processes in a command group, in a pipeline, executed sequentially under certain conditions.

[Dan Douglas]

Hello, I ran across a behavior I don't quite understand while playing with 
redirecting process substitution's pipes.  Specifically, it seems that a 
compound command containing a list of background jobs is affected by whether 
or not the compound command is part of a pipeline. Either the commands don't 
get backgrounded; they do, but somehow block and aren't executed concurrently; 
or some side-effect of the redirection order not being applied as I expect is 
making it appear so.

The below script contains several testcases. I'm borrowing the named pipe of a 
process substitution to share output between multiple background readers which 
should be getting approximately interleaved data if both are running at the 
same time. The first example does exactly that, but changing the output from a 
redirected process substitution to a pipe (into read) in the second and third 
examples causes one of the readers to consume everything. Mysteriously, the 
fourth and fifth examples then go back to working as the first even with the 
pipe simply by moving the redirection inside the command group. ( Unless I'm 
misunderstanding how this works... the redirect to the pipe happens first, 
then redirects to the command group, then everything within the command group 
starts out with their stdio pointed to the same place, and the extra redirects 
like {x}< should be redundant. ) The last example is different and probably 
unrelated. Not sure whether this is a bug or desired behavior for the named 
pipe to go away before it's used.

Here's a gist with nice highlighting: https://gist.github.com/1255778

I asked this over on the unix.com forums a few days ago but unsurprisingly 
didn't get much attention.

#!/usr/bin/env bash

set +m -f                          # Was doing some unsafe wordsplitting.
shopt -s lastpipe                  # Not really necessary. No effect afaict.

declare -i ex=1
exincr() { echo "ex $((ex++)):"; }

zeros() { printf "%.s0" {1..30}; } # Each example shares a stream of zeros from 
a pipe as input.

f() {
    local -i x y
    while read -rN1 "x[y++]"; do
        printf '%d ' "${1}" >&2    # keep track of which job this is.
    done
    printf "${#x[@]} "             # Print the total number of reads by each 
job.
}

g() {                              # Used in ex 6
    f 1 <${1} &
    f 2 <${1}
}

# This works as I expect, f is backgrounded and two readers of one pipe each 
get about half the input:
exincr # 1

read -ra x < <({ f 1 & f 2; } < <(zeros))
printf '%b\n' "\n${x[@]}\n"

# Equivalent to above, except with piped output. Now f is not backgrounded. One 
reader consumes all the input:
exincr # 2

{ f 1 & f 2; } < <(zeros) | {
    read -ra x
    printf '%b\n' "\n${x[@]}\n"
}

# Same as above. Different pipe configuration, but same behavior.
exincr # 3

zeros | { f 1 & f 2; } | read -ra x
printf '%b\n' "\n${x[@]}\n"

# Like the above two examples, pipe the output, except save the FD to $x and 
redirect to each f individually. Strangely, it now 
behaves as the first example again.
exincr # 4

{ f 1 <&$z & f 2 <&$z; } {z}< <(zeros) | {
    read -ra x
    printf '%b\n' "\n${x[@]}\n"
}

# Additionally, the first command in the list to be explicitly redirected to 
the proper input, and all subsequent list members, are 
executed asynchronously as expected. Again, no idea why.
exincr # 5

{ f 1 & f 2 <&$z & f 3 & } {z}< <(zeros) <&$z | {
    read -ra x
    printf '%b\n' "\n${x[@]}\n"
}

# In this version, the name of the pipe is passed to g and then individually 
redirected to f, but it doesn't work. No matter how 
much data it gets, the pipe is closed before the second call to f sees it:
exincr # 6

read -ra x < <(g <(zeros))
wait
printf '%b\n' "\n${x[@]}\n"

# vim: set fenc=utf-8 ff=unix ts=4 sts=4 sw=4 ft=sh nowrap et:

And the output:

$ schedtool -I -e ./pipefork
ex 1:
2 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2
15
17

ex 2:
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
1
31

ex 3:
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
1
31

ex 4:
1 2 1 2 2 1 2 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2
15
17

ex 5:
2 2 3 2 3 2 3 2 3 2 3 2 3 3 2 3 3 3 2 3 3 2 3 2 3 3 2 3 3 2
1
18
14

ex 6:
./pipefork: line 21: /dev/fd/63: No such file or directory
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
31

...And my Bash info just in case:

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: x86_64-pc-linux-gnu-gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -
DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gnu' -
DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL -
DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib  -
DDEFAULT_PATH_VALUE='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
 
-DSTANDARD_UTILS_PATH='/bin:/usr/bin:/sbin:/usr/sbin' -
DSYS_BASHRC='/etc/bash/bashrc' -DSYS_BASH_LOGOUT='/etc/bash/bash_logout' -
DNON_INTERACTIVE_LOGIN_SHELLS -DSSH_SOURCE_BASHRC -march=native -Ofast -ggdb -
mmmx -floop-interchange -floop-strip-mine -floop-block -pipe
uname output: Linux smorgbox 3.0.4-pf+ #36 SMP PREEMPT Sat Sep 24 16:47:49 CDT 
2011 x86_64 Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz GenuineIntel GNU/Linux
Machine Type: x86_64-pc-linux-gnu

Bash Version: 4.2
Patch Level: 10
Release Status: release

Thanks for having a look!
-Dan Douglas

signature.asc
Description: This is a digitally signed message part.