Re: [pdf-devel] Patch for the Flate filter

[gerel]

 > Date: Fri, 03 Oct 2008 03:47:41 +0200
 > From: address@hidden
 > 
 >    My question comes down to this:
 > 
 >    Is it correct to return PDF_ENOUTPUT even if the apply function is called 
 > with
 >    finish_p=TRUE ? If the answer is yes, then the logic I mentioned above is 
 > being
 >    considered in the architecture, and the problem has been solved. 
 > Otherwise, the
 >    bug is still there.
 > 
 > No, currently it is not possible to do that. What happens if you
 > decide to "finish" the encoding with an empty input buffer? How many
 > data would that generate? What is the concrete problem with the flate
 > encoder?

The problem is not the flate encoder in particular, it's common sense, this is 
why:

Encoders tipically allocate an internal buffer for compressing data. This
internal buffer is big enough such that the encoder can make use of greater
patterns in data and encode better.
Now, say that a given encoder E uses an internal buffer of 256 Kbytes and
suppose that the user provided his 6 _last_ Kbytes. At this point say that
the encoder's internal buffer has 200 Kbytes free. So,

        total_internal_buf      = 256 KB
        free_internal_buf       = 200 KB
        user_total_write_size   = 46  KB
        user_last_write_size    = 10  KB

Since the encoder's idea is to fill those 256k bytes (for better compression),
the user has to tell it "ok I won't send more data so compress with what I gave
you (46 + 10 = 56 KB) so far".
Now, at this point, the encoder starts spitting out the encoded data, say this
encoder compresses the total 56 KB you gave it to 20 KB.
Finally so, the issue is that you can't predict that the encoder will spits 20
compressed KB, as you can't predict the data the user gave you. Welcome to the
streaming world :-)

This is why, you can't tell a filter when to stop writing. What you _can_ do, is
make sure that as long as the filter writes, someone on the other side is going
to have to read it, or in the case of the encoder, you'll try to write it into
the backend.

Now back to the questions,

 > What happens if you decide to "finish" the encoding with an empty input 
 > buffer?

Well, that's the idea in the end, a finish for an encoder is, flush
internal_buffer + headers.

> How many data would that generate?

As mentioned above, you (the filter/encoder developer) don't know, only the user
could predict that by knowing how much he'll write in total and providing a big
enough backend. (not our issue at all)

> What is the concrete problem with the flate encoder?

I hope what I said above helps on this one. :-)

cheers

-gerel