Re: Adding slice-by-4 and slice-by-8 to CRC32

[Sam Russell]

I built from HEAD, named it gzip_vanilla, rebuilt with my CRC code and named it gzip_8_slice. Input file is a 115MB file gzipped (default settings) to 61M

sam@DESKTOP-R64B0KJ:~/gziptest$ time ../code/gzip/gzip_8_slice -k -d -c large_file.gz > /dev/null

real    0m0.307s
user    0m0.298s
sys     0m0.000s
sam@DESKTOP-R64B0KJ:~/gziptest$ time ../code/gzip/gzip_8_slice -k -d -c large_file.gz > /dev/null

real    0m0.312s
user    0m0.310s
sys     0m0.000s
sam@DESKTOP-R64B0KJ:~/gziptest$ time ../code/gzip/gzip_8_slice -k -d -c large_file.gz > /dev/null

real    0m0.306s
user    0m0.307s
sys     0m0.000s
sam@DESKTOP-R64B0KJ:~/gziptest$ time ../code/gzip/gzip_8_slice -k -d -c large_file.gz > /dev/null

real    0m0.319s
user    0m0.316s
sys     0m0.000s
sam@DESKTOP-R64B0KJ:~/gziptest$ time ../code/gzip/gzip_vanilla -k -d -c large_file.gz > /dev/null

real    0m0.485s
user    0m0.484s
sys     0m0.000s
sam@DESKTOP-R64B0KJ:~/gziptest$ time ../code/gzip/gzip_vanilla -k -d -c large_file.gz > /dev/null

real    0m0.498s
user    0m0.476s
sys     0m0.020s
sam@DESKTOP-R64B0KJ:~/gziptest$ time ../code/gzip/gzip_vanilla -k -d -c large_file.gz > /dev/null

real    0m0.488s
user    0m0.478s
sys     0m0.010s

Looks to be about a ~35% reduction in time

On Mon, 14 Oct 2024 at 17:44, Bruno Haible <bruno@clisp.org> wrote:

Sam Russell wrote:
> As for your question on speed, I noticed between zstd (which uses zlib as a
> backend) and gzip there seems to be an improvement of maybe 30-40% for
> decompressing a 100MB file (part of this is due to multithreading though),

If you compare two scenarios which differ in 4 aspects:
  - different general codebase,
  - different CRC32 implementation,
  - different multithreading,
  - code compared with -fPIC (shared library) vs. without -fPIC (gzip program),
you cannot draw any conclusion. In order to make a valid point, you need to
vary _one_ aspect only.

> and gprof shows the CRC calculation being maybe 40-50% of the CPU cycles

Don't blindly trust gprof figures. gprof is sensitive to a function's size
and invocation frequency. There are better profilers [1], but in this case
what I asked for is not a profiling but a comparison of two scenarios that
differ in 1 aspect, each compiled in the usual way ("gcc -O2", with a recent
gcc, on x86_64 or arm64).

Bruno

[1] https://gitlab.com/ghwiki/gnow-how/-/wikis/Profiling