[Openexr-devel] DWA compression: fails due to insufficient buffer size

[Vasil Minkov]

Hello,

While tracking an error report I found an issue with the DWA compression 
- it fails to compress certain input data.
The problem is easily reproduced on attempt to store a small (8x8 
pixels) single-channel (Luma only) EXR image using DWA compression.
An exception "Data compression (zlib) failed." is thrown in such case. 
The problem also occurs when compressing small and noisy RGB tiles.

It seems that the problem is in ImfDwaCompressor.cpp, in the following 
code fragment:

                ......
                uLongf destLen = (uLongf)
                    (2 * (*totalAcUncompressedCount) * sizeof (unsigned 
short));

                if (Z_OK != ::compress2
                                ((Bytef *)outDataPtr,
&destLen,
                                 (Bytef *)_packedAcBuffer,
                                 (uLong)(*totalAcUncompressedCount
                                                * sizeof (unsigned short)),
                                 9))
                ......

Under certain circumstances compress2() output size may exceed its input 
size, as specified in zlib's docs.
Here is my fix of the problem:

                ......
                uLongf destLen = (uLongf)
                    (2 * (*totalAcUncompressedCount) * sizeof (unsigned 
short));
                // compress2 requirement:
                //   "Upon entry, destLen is the total size of the
                //   destination buffer, which must be at least 0.1% larger
                //   than sourceLen plus 12 bytes"
                // Fulfill the above requirement in integer arithmetic
                destLen=((Int64)destLen*1001 + 1000)/1000 + 12;
                if (Z_OK != ::compress2
                                ((Bytef *)outDataPtr,
&destLen,
                                 (Bytef *)_packedAcBuffer,
                                 (uLong)(*totalAcUncompressedCount
                                                * sizeof (unsigned short)),
                                 9))
                ......

zlib's ::compressBound() could also be used to calculate the buffer size.

Regards,

--
Vasil Minkov

VRay for 3ds Max developer