Re: Statement about the maintenance of lzip

[Steffen Nurpmeso]

Hello Antonio.

Antonio Diaz Diaz wrote in
 <66195D71.4090601@gnu.org>:
 |Steffen Nurpmeso wrote:
 |> How about malloc hookability?
 |
 |https://developers.redhat.com/articles/2021/08/25/securing-malloc-glibc-\
 |why-malloc-hooks-had-go
 |"The key misfeature of the debugging hooks, though, was their presence as 
 |unprotected function pointers that were guaranteed to be executed at 
 |specific events. This made the hooks an easy exploit primitive in 
 |practically every program that ran on Linux distributions. A trivial \
 |search 
 |for __malloc_hook "house of" turns up a long list of exploit methods that 
 |use the hooks as either an intermediate step or the final goal for \
 |the exploit."

I do not get that.  Once libressl was forked from openssl (around
2015 maybe) it was one of the first commits to remove the malloc
hooks.  I just used them (openssl) to make the MUA i maintain
a bit more clean.  So that was much, much earlier.
This is the C library.
In the meantime openssl themselves removed them i think; this is
a real pity, imho.
You need debugger attachability to a SET*ID program for this to
work out, or you need to be root yourself.
Or you need to be a linked-in library, or whatever.
Can't you do almost anything in practice in these circumstances,
anyway?
If i read that, it seems more about speed or thread-awareness,
their fault, thus.

 |I prefer to use valgrind. Someday someone will discover that preloading 
 |malloc can also be exploited.

Yes.  valgrind never worked for me btw, it had file descriptor dup
problems (in that my tests then suddenly produced data twice or so
because; just recently had this problem again because Open/NetBSD
do not support fflush() for o_rdonly streams meaning sync position
with descriptor, but otoh optimize so that concurrent writes via
another descriptor "remain undetected"; but this is a different
thing of course).  I admire coverity.

I love those libraries which allow such, you know, i can make them
use dedicated memory.  Even per-object-instance.
Like "i know XY and want YZ".
Like lzip states it needs XY bytes of memory, and i can ship that
from within a nice location.
malloc is not signal safe.
The entire C library is not.
For example.
Anyhow the above i do not get.
My memory pool, for example, allowed hooking at program startup;
these hooks only execute when the pool needs to interact with
large continuous arenas from the operating system, which occurs
scarcely only

      pub static void hook(   void *(*_Alloc)(uir _sz, MemoryType _type),
                              boolean (*_FreeOK)(void *_mem, uir _sz),
                              boolean _collectfree,
                              void (*_Free)(void *_mem, uir _sz,
                                                      MemoryType _type),
                              void (*_Madv)(void *_mem, uir _sz),
                              boolean _usemadv);

Like that one can create a program with one large BSS section
which is then used to serve the program ifself.
Granted, likely only in cases where the C library or other
non-hookable libraries are not used, but >20 years ago this was
quite common for many things.  And worked out nicely.

Ie LZ_(de)?compress_open_malloc()? :-)
Ach, that would be nice..  But you do not need to treat this
seriously.

 |> git (at least an automated conversion to Savannah / xy)?
 |
 |Consider lzip as a work of literature. I just won't reveal my writing \
 |habits 
 |to the world.

Of course, this is all up to your personal wishes.  Thomas Dickey
(ncurses, lynx, vile, xterm, etc.) only provides development
snapshots via git, ie

  https://github.com/ThomasDickey/lynx-snapshots.git

  (tag: refs/tags/v2-9-0f, ..) snapshot of project "lynx", label v2-9-0f
  (tag: refs/tags/v2-9-0e) snapshot of project "lynx", label v2-9-0e

Likely all-automatic.  Sometimes there are six months and more
(vile, at least) in between updates.
Just in case this option never occurred to you and would be seen
as an option.

 |> I think the BSDs refrain because of GPL; at least it is v2 not v3,
 |> but still; zstd instead (i think: "then, later") changed to
 |> BSD-style OR GPLv2 saying "You may select, at your option, one of
 |> the above-listed licenses".
 |
 |There is a fully functional permissively licensed implementation of \
 |the lzip 
 |data compressor: http://www.nongnu.org/lzip/pdlzip.html
 |
 |Pdlzip is free software. The (de)compression code is in the public domain. 
 |The rest is under a 2-clause BSD license.

I have not seen this yet, i use plzip.  Oh!
Quite a lot of different things to move forward.

 |Lzlib is free software distributed under a 2-clause BSD license.

Ah.  Oh!!  I had only seen COPYING.GPL and neither looked at
COPYING nor lzip.h!  Actually i did look into minilzip.c, which is
GPLv2.  (I asked my CRUX Linux to not only --enable-shared but
also install this in addition, as it is compiled anyway!)
That is really great!!  It should be noted more prominently that
this is not, actually, GPL-licensed sofware!

 |> While CRC-32 is ok, i guess people (including me) doubt its
 |> viability for long-term archiving, especially when compared with
 |> other algorithms.  It is not so terrible as years ago, since most
 |> people surely have lots of copies, and the filesystems use
 |> checksumming.  But as a standalone archive, CRC-32 fails badly,
 |> for example smhash says "insecure, 8590x collisions, distrib,
 |> PerlinNoise":
 |
 |The tests performed by smhasher are 100% unrelated to error detection in a 
 |decompressor context. CRC32 is probably optimal to detect errors in lzip 

This is true.

 |members. See 
 |http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance

This document is not part of the distribution.  At least this
section.

 |"Lzip, like gzip and bzip2, uses a CRC32 to check the integrity of the 
 |decompressed data because it provides optimal accuracy in the detection of 
 |errors up to a compressed size of about 16 GiB, a size larger than that of 
 |most files. In the case of lzip, the additional detection capability \
 |of the 
 |decompressor reduces the probability of undetected errors several million 
 |times more, resulting in a combined integrity checking optimally accurate 
 |for any member size produced by lzip."
 |
 |See also http://www.nongnu.org/lzip/safety_of_the_lzip_format.html#lzma_crc
 |'4.1 Interaction between LZMA compression and CRC32' and '7 Conclusions':
 |
 |"After 14 years of testing, the MTBF of lzip can only be estimated because 
 |not even one false negative has ever been observed. If one were to 
 |continuously decompress corrupt lzip files of about one megabyte in \
 |size (10 
 |decompressions per second), each of them containing the kind of corruption 
 |most difficult to detect (one random bit flip), then a false negative \
 |would 
 |be expected to happen every 694 million years."

Well this is very nice mathematics.  But, and please, i know
CRC-32 is used for decades for digesting in a lot of standards,
etc, and MD5 and SHA-1 are both "broken" cryptographically but
are also still used for checksumming.  Mathematically i could
imagine xxhash32 would be a bit better in the same 32-bit.

 |Best regards,
 |Antonio.

Thank you very much!

 --End of <66195D71.4090601@gnu.org>

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)