Re: [Bug-tar] Multi-threaded tar

[J.R. Heisey]

Hi Joerg,

I ran some tests and I'm seeing anywhere from 9.38% to 33 % performance improvement for various activities.

I'm calculating the percentage based to the best test run for each activity between tar and star.

The time durations are in mm:ss format.

The computer I was using may have automated processes startup periodically so I discount the slow running outliers.

Is this the performance improvement that you would expect with star?

I hope the formatting holds. :)

	create	Percent	compress bzip2	Percent	compress pbzip2	Percent	decompress bzip2	Percent	decompress pbzip2	Percent
tar	1:06		2:24		1:31		1:14		0:56
	1:05		2:28		2:10		1:04		0:56
	1:15		2:20		1:35		1:10		0:58
					1:34
star	2:10		4:24		1:15		0:58		0:46
	1:11		2:08		1:30		2:24		0:51
	0:43		2:04		1:32		2:29		0:52
	1:01
using minimums		33.85%		11.43%		17.58%		9.38%		17.86%
time star  -xj f=testdata-7.13.0-1.pb.star.gz -C ./sources
time star -compress-program pbzip2 -x f=testdata-7.13.0-1.pb.star.gz -C ./sources
time star -cz f=./testdata-7.13.0-1.star.gz -C ../util-build/testdata-7.13.0-1/sources .
time star -compress-program pbzip2 -c f=./testdata-7.13.0-1.pb.star.gz -C ../util-build/testdata-7.13.0-1/sources .
time star -c f=./testdata-7.13.0-1.star.gz -C ../util-build/testdata-7.13.0-1/sources .
time tar -xzf ./testdata-7.13.0-1.tar.gz .
time tar --use-compress-program pbzip2 -xf ./testdata-7.13.0-1.pbzip2.tar.gz .
time tar -czf ./testdata-7.13.0-1.pb.2.tar.gz -C ../util-build/testdata-7.13.0-1 ./sources/
time tar --use-compress-program pbzip2 -cf ./testdata-7.13.0-1.pb.2.tar.gz -C ../util-build/testdata-7.13.0-1 ./sources/
time tar -cf ./testdata-7.13.0-1.tar -C ../util-build/testdata-7.13.0-1 ./sources/

Thanks,

J.R.

On Tue, Jul 31, 2018 at 3:23 AM, Joerg Schilling <address@hidden> wrote:

"J.R. Heisey" <address@hidden> wrote:

> Greetings,
>
> I was trying to make some processes faster which use tar a bit on large archives.
>
> I was able to use the command line options --use-compress-program pbzip2 and that helped a lot.
>
> I was wondering if anyone has experimented using pthreads in the tar implementation. I don't see any references in version 1.30 of the source or readme I downloaded. Anyone discussed a strategy for using pthreads in previous postings?

Star forks and runs two processes since approx. 30 years.

The first process is for the filesystem I/O and the second process is for the
archive I/O. between both, there is a ring buffer with configurable size that
decouples both tasks. This helps to make star very fast.

Implementing something like this in a TAR implementation that does not yet have
support for it is highly complex.

Jörg

--
 EMail:address@hidden                    (home) Jörg Schilling D-13353 Berlin
    address@hiddenfraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'