parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

--pipepart implemented


From: Ole Tange
Subject: --pipepart implemented
Date: Wed, 9 Apr 2014 05:22:05 +0200

I have now implemented ---pipepart. It works like --pipe except that
it is way more efficient and works only on physical files. It does not
(and will never) support -N or -L as we cannot tell what line we are
on if we seek into the middle of the file.

    parallel -a bigfile1 --pipepart --block 1g wc -c

This works similar to:

    cat bigfile1 | parallel --pipe --block 1g wc -c

You can have multiple bigfiles:

    parallel -a bigfile1 -a bigfile2 --pipepart --block 1g wc -c

This will work similar to:

    cat bigfile1 bigfile2 | parallel --pipe --block 1g wc -c

Performance on my 8 core machine is a kickass 3.3 GB/s(!) compared to
350 MB/s using --pipe.

I am sure there are bugs, so start hunting.

    git clone git://git.savannah.gnu.org/parallel.git


/Ole



reply via email to

[Prev in Thread] Current Thread [Next in Thread]