Possible to use process substitution a variable number of times?

Question

Hello Archers,
I have a script like so:
#! /bin/bash
( readlink /proc/$$/fd/0 | grep -q "^pipe:" ) || ( file $(readlink /proc/$$/fd/0) | grep -q "character special" )
cat | tee >(awk 'NR % 2 == 0') >(awk '(NR+1) % 2 == 0')
Well, it's pretty trivial at the moment, but the underlying goal is to split the input, so it can be processed in parallel on multiple cores. If I understand correctly, the above code will split the input across exactly two cores. Does there exist some way to put the process substitution (i.e. ">(awk ' ... ')")  in a loop?
Before trying to increase the number of cores, I first tried to see whether I could loop just two times with,
cat | tee $(for i in 1 2; do >(awk -vn="$i" '(NR + $n) % 2 == 0'); done)
but that didn't work.
It also occurred to me that maybe I'm presenting an X-Y problem. I did also consider using the split command, but as far as I can tell, it will only dump its output to files; it's no good for piping or process substitution. Any other ideas that would avoid needing to tee across a variable number of process substitutions?
Naturally, I tried a little Googlemancy, but only came up with answers to newbie questions, or tutorials aimed at newbies.
Thank you for your consideration.
Last edited by /dev/zero (2012-03-26 07:16:25)

Answer 1

Or if you want to avoid named pipes, since during development they tend to create lots of files, and the script has to clean them up afterwards, take a look at the 'coproc' feature bash has since version 4
Though, remember that 'man bash' under BUGS says, there may only be one active coproc at a time.
According to this it's not really such a big deal though.
And it won't kill the started coprocesses or anything if you start another process...
Here's a sample session: (the spaces are important btw. )
wry:~/ $ bash [19:02:43]
wry@blubmb:~$ coproc c1 ( cat )
[1] 4129
wry@blubmb:~$ coproc c2 ( cat )
bash: warning: execute_coproc: coproc [4129:c1] still exists
[2] 4130
wry@blubmb:~$ echo $c1_PID $c2_PID -- ${c1[@]} ${c2[@]}
4129 4130 -- 63 60 62 58
wry@blubmb:~$ echo aaac1 >&${c1[1]}
wry@blubmb:~$ echo bbbc2 >&${c2[1]}
wry@blubmb:~$ read -u ${c1[0]} line1
wry@blubmb:~$ read -u ${c2[0]} line2
wry@blubmb:~$ echo $line1 and $line2
aaac1 and bbbc2
wry@blubmb:~$ kill $c1_PID $c2_PID
wry@blubmb:~$
[1]- Terminated coproc c1 ( cat )
[2]+ Terminated coproc c2 ( cat )
wry@blubmb:~$ exit
exit
Just keep in mind that the filedescriptors you get for a coproc won't be available in subshells you execute with &
Eg. you cannot do
coproc X
( use X ) &
Links: http://wiki.bash-hackers.org/syntax/keywords/coproc