[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: newbie ?
From: |
Ole Tange |
Subject: |
Re: newbie ? |
Date: |
Fri, 7 Apr 2017 00:04:07 +0200 |
On Tue, Apr 4, 2017 at 10:22 PM, ed bierly <ebierly@gmail.com> wrote:
> my goal is to use this stdout so i think i need --pipe in order to tee out a
> single stream
> pbzip2 -cd 25Gzipped.bz2|parallel --pipe ...
>
> the awk program would now be like this
> {if(a==$1)print $0>$1}
What you are looking for is '--pipe --tee':
seq 3 | parallel --tee --pipe awk -v a={} -f a.awk ::: 2 3
It is exactly made for your situation where the input is slow to generate.
--tee is very new and is in alpha test, so please give feedback.
There are some limitations:
Every job uses 4 file handles, so on a normal GNU/Linux system you
will be able to run 250 jobs in parallel. This can be raised with
`ulimit -n 65535` giving 16300 jobs.
All jobs will be run in parallel. This is due to the input not being
saved, but sent directly to the jobs.
/Ole