|
From: | ed bierly |
Subject: | Re: newbie ? |
Date: | Thu, 6 Apr 2017 22:42:38 -0400 |
On Tue, Apr 4, 2017 at 10:22 PM, ed bierly <ebierly@gmail.com> wrote:
> my goal is to use this stdout so i think i need --pipe in order to tee out a
> single stream
> pbzip2 -cd 25Gzipped.bz2|parallel --pipe ...
>
> the awk program would now be like this
> {if(a==$1)print $0>$1}
What you are looking for is '--pipe --tee':
seq 3 | parallel --tee --pipe awk -v a={} -f a.awk ::: 2 3
It is exactly made for your situation where the input is slow to generate.
--tee is very new and is in alpha test, so please give feedback.
There are some limitations:
Every job uses 4 file handles, so on a normal GNU/Linux system you
will be able to run 250 jobs in parallel. This can be raised with
`ulimit -n 65535` giving 16300 jobs.
All jobs will be run in parallel. This is due to the input not being
saved, but sent directly to the jobs.
/Ole
[Prev in Thread] | Current Thread | [Next in Thread] |