[Lzip-bug] lzlib: need help with LZ_compress_sync_flush()

[Tino Lange]

Hi Antonio, hi List-Members,

I'm currently playing with lzlib and I really do like the interface and
the results very much! Thanks so far!

Now as I'm playing with the LZ_compress_sync_flush() function I ran into
problems. Hopefully you're able to help me with that?

[ Hmmmm, maybe this is no bug, but pure API/Documentation
misunderstanding on my side, in that case please forgive my sending to
lzip-bug (I wasn't able to find any lzip-users or similar mailing list).
Please feel free to redirect me if I'm abusing this list with my post. ]

Here we are:

Scenario:

-> compressing part opens a compressor and compresses independent chunks
of data (each of different size). After each chunk if flushes with
LZ_compress_sync_flush() and sends all data it can read to the
decompressing counterpart.

-> decompressing part opens a decompressor and receives data in such
chunks from compressing part and tries to decompress them. Of course it
expects that each chunk is decompressable since it was flushed on the
other side.

Problem:

I can't get this working with flush and one kept
compressor/decompressor. It seems I'm missing some kind of
"LZ_decompress_sync_flush()" on the receiving side to notify the
decompressor that one chunk is finished? Analogous to the sending side?

Please see attached source code as example, it compiles with
g++ -I $PATH_TO_LZLIB/include -L $PATH_TO_LZLIB/lib -o test test.cc -llz

If I change the LZ_compress_sync_flush() call to LZ_compress_finish()
and use a new initialized compressor/decompressor on each invocation on
compress() / decompress() the whole thing works of course.

What am I missing to get it running when keeping the
compressor/decompressor instances and using flush? Or did I really
stumble across some kind of bug?

Thanks for any help and best regards

Tino

#include <iostream>
#include <vector>
#include <cassert>
#include <limits>
#include <stdexcept>

#include <boost/lexical_cast.hpp>

#include <lzlib.h>

////////////////////////////////////////////////////

struct lzma_options
{
    int dictionary_size;          // 4KiB..512MiB
    int match_len_limit;          // 5..273
};

static const lzma_options option_mapping[] = {
    { 1 << 20,  10 },           // -1
    { 1 << 20,  12 },           // -2
    { 1 << 20,  17 },           // -3
    { 1 << 21,  26 },           // -4
    { 1 << 22,  44 },           // -5
    { 1 << 23,  80 },           // -6
    { 1 << 24, 108 },           // -7
    { 1 << 24, 163 },           // -8
    { 1 << 25, 273 }            // -9
};

////////////////////////////////////////////////////

static void* compressor = 0;

static void init_compressor(unsigned int compression_level = 6)
{
        assert(compression_level < sizeof(option_mapping));

    const lzma_options compressor_options = option_mapping[compression_level];

    if (compressor)
                LZ_compress_close(compressor);
    compressor = LZ_compress_open(compressor_options.dictionary_size,
                                                                  
compressor_options.match_len_limit,
                                                                  
std::numeric_limits<long long>::max());
        
    const LZ_errno lz_errno = LZ_compress_errno(compressor);
    if (!compressor || lz_errno != LZ_ok) {
                LZ_compress_close(compressor);
                if (lz_errno == LZ_mem_error)
                        throw std::bad_alloc();
                throw std::runtime_error("lzlib error " + 
boost::lexical_cast<std::string>(lz_errno));
    }
}

////////////////////////////////////////////////////

static void* decompressor = 0;

static void init_decompressor()
{
    if (decompressor)
                LZ_decompress_close(decompressor);    
    decompressor = LZ_decompress_open();
    const LZ_errno lz_errno = LZ_decompress_errno(decompressor);
    if (!decompressor || lz_errno != LZ_ok) {
                LZ_decompress_close(decompressor);
                if (lz_errno == LZ_mem_error)
                        throw std::bad_alloc();
                throw std::runtime_error("lzlib error " + 
boost::lexical_cast<std::string>(lz_errno));
    }
}

////////////////////////////////////////////////////

static void compress(const std::string& msg, std::vector<unsigned char>& 
out_buffer)
{
    // in buffer / msg (uncompressed)
    unsigned long msg_position = 0;
    
    // out buffer (compressed)
    out_buffer.resize(((msg.size() / 5) / (64 * 1024) + 1 ) * (64 * 1024) + (64 
* 1024));
        //std::cerr << "compress() initialized compression out_buffer to "
        //          << out_buffer.size() << std::endl;
    unsigned long out_buffer_position = 0;
    
    // do the compression
    while (true) {
        
                if (msg_position < msg.size()) {
                        const int len = LZ_compress_write(compressor,
                                                                                
          reinterpret_cast<uint8_t*>(const_cast<char*>(&msg[msg_position])),
                                                                                
          msg.size() - msg_position);

                        std::cerr << "LZ_compress_write(), pos: " << 
msg_position
                                          << ", len: " << msg.size() - 
msg_position
                                          << " -> len: " << len << std::endl;

                        if (len <= 0) {
                                const LZ_errno lz_errno = 
LZ_compress_errno(compressor);
                                throw std::runtime_error("lzlib error 
(compress_write), len="
                                                                                
 + boost::lexical_cast<std::string>(len)
                                                                                
 + ", lz_errno="
                                                                                
 + boost::lexical_cast<std::string>(lz_errno)
                                                                                
 + " (msg: pos/size "
                                                                                
 + boost::lexical_cast<std::string>(msg_position) + "/"
                                                                                
 + boost::lexical_cast<std::string>(msg.size()) + ")");
                        }
            
                        msg_position += len;
            
                        if (msg_position >= msg.size()) {
                                LZ_compress_sync_flush(compressor);
                                std::cerr << "LZ_compress_flush() " << 
std::endl;
                        }
                }
        
                if ((out_buffer.size() - out_buffer_position) < 64 * 1024 /* 64 
kB */) {
                        out_buffer.resize(out_buffer.size() + 64 * 1024);
                        //std::cerr << "compress() extended compression 
out_buffer by 64 kB to "
                        //          << out_buffer.size() << std::endl;
                }
                
                const int len = LZ_compress_read(compressor, 
reinterpret_cast<uint8_t*>(&out_buffer[out_buffer_position]),
                                                                                
 out_buffer.size() - out_buffer_position);

                std::cerr << "LZ_compress_read(), pos: " << out_buffer_position
                                  << ", len: " << out_buffer.size() - 
out_buffer_position
                                  << " -> len: " << len << std::endl;

                if (len < 0) {
                        const LZ_errno lz_errno = LZ_compress_errno(compressor);
                        throw std::runtime_error("lzlib error (compress_read), 
len="
                                                                         + 
boost::lexical_cast<std::string>(len)
                                                                         + ", 
lz_errno="
                                                                         + 
boost::lexical_cast<std::string>(lz_errno)
                                                                         + " 
(out_buffer: pos/size "
                                                                         + 
boost::lexical_cast<std::string>(out_buffer_position) + "/"
                                                                         + 
boost::lexical_cast<std::string>(out_buffer.size()) + ")");
                }
                
                if (len == 0 && msg_position >= msg.size())
                        break;
        
                out_buffer_position += len;
    }
    
    out_buffer.resize(out_buffer_position);

        std::cerr << "compress() sucessfully compressed string of size " << 
msg.size()
                          << " to " << out_buffer.size() << std::endl;
}

////////////////////////////////////////////////////

void decompress(const std::vector<unsigned char>& in_buffer, std::string& msg) 
{
    // in buffer (compressed)
    unsigned long in_buffer_position = 0;

    // out buffer / msg (uncompressed)
    msg.resize(((in_buffer.size() * 5) / (64 * 1024) + 1 ) * (64 * 1024) + (64 
* 1024));
        //std::cerr << "decompress() initialized decompression out_buffer (msg) 
to "
        //          << msg.size() << std::endl;
    unsigned long msg_position = 0;

    // do the decompression
    while (true) {
        
                if (in_buffer_position < in_buffer.size()) {
                        const int len = LZ_decompress_write(decompressor,
                                                                                
                reinterpret_cast<uint8_t*>(const_cast<unsigned 
char*>(&in_buffer[in_buffer_position])),
                                                                                
                in_buffer.size() - in_buffer_position);

                        std::cerr << "LZ_decompress_write(), pos: " << 
in_buffer_position
                                          << ", len: " << in_buffer.size() - 
in_buffer_position
                                          << " -> len: " << len << std::endl;

                        if (len <= 0) {
                                const LZ_errno lz_errno = 
LZ_decompress_errno(decompressor);
                                throw std::runtime_error("lzlib error 
(decompress_write), len="
                                                                                
 + boost::lexical_cast<std::string>(len)
                                                                                
 + ", lz_errno="
                                                                                
 + boost::lexical_cast<std::string>(lz_errno)
                                                                                
 + " (in_buffer: pos/size "
                                                                                
 + boost::lexical_cast<std::string>(in_buffer_position) + "/"
                                                                                
 + boost::lexical_cast<std::string>(in_buffer.size()) + ")");
                        }
            
                        in_buffer_position += len;
                }
        
                if ((msg.size() - msg_position) < 64 * 1024 /* 64 kB */) {
                        msg.resize(msg.size() + 64 * 1024);
                        //std::cerr << "decompress() extended decompression 
out_buffer (msg) by 64 kB to "
                        //          << msg.size() << std::endl;
                }
        
                const int len = LZ_decompress_read(decompressor, 
reinterpret_cast<uint8_t*>(&msg[msg_position]),
                                                                                
   msg.size() - msg_position);

                std::cerr << "LZ_decompress_read(), pos: " << msg_position
                                  << ", len: " << msg.size() - msg_position
                                  << " -> len: " << len << std::endl;

                if (len < 0) {
                        const LZ_errno lz_errno = 
LZ_decompress_errno(decompressor);
                        throw std::runtime_error("lzlib error 
(decompress_read), len="
                                                                         + 
boost::lexical_cast<std::string>(len)
                                                                         + ", 
lz_errno="
                                                                         + 
boost::lexical_cast<std::string>(lz_errno)
                                                                         + " 
(out_buffer: pos/size "
                                                                         + 
boost::lexical_cast<std::string>(msg_position) + "/"
                                                                         + 
boost::lexical_cast<std::string>(msg.size()) + ")");
                }
        
                
                if (len == 0 && in_buffer_position >= in_buffer.size())
                        break;

                msg_position += len;
    }
    
        msg.resize(msg_position);

        std::cerr << "decompress() sucessfully decompressed string"
                          << " from " << in_buffer.size() << " to " << 
msg_position << std::endl;
}

////////////////////////////////////////////////////

int main()
{
    if (LZ_version()[0] != LZ_version_string[0])
                throw std::runtime_error("lzlib error, incompatible library 
version");

    init_compressor();
    init_decompressor();
        
        std::vector<std::string> msgs;
        msgs.push_back("This is the 1st message.");
        msgs.push_back("This is the 2nd message.");
        msgs.push_back("This is the 3rd message.");

        for (std::vector<std::string>::size_type i = 0; i < msgs.size(); ++i) {
                std::cout << "\n******************************\n\n";
                std::cout << "E: '" << msgs[i] << "'" << std::endl;
                std::vector<unsigned char> data;
                compress(msgs[i], data);
                std::string msg;
                decompress(data, msg);
                std::cout << "D: '" << msg << "'" << std::endl;
        }

        return 0;
}