Re: [pdf-devel] [PATCH] Crypt filters

[jemarch]

Hi David.

   I include a patch for the crypt filters support. This patch adds:
     - AESV2 encoder/decoder filters
     - V2 encoder/decoder filters
     - MD5 encoder filter
     - Fix several bugs in crypt module

   It is not complete. Although it works, it needs love. :)

In the md5 encoder:

  if (finish_p == PDF_TRUE)
    {
      /* Once we have collected all input, then we can go on. */
      pdf_size_t cache_size;
      pdf_size_t out_size;

      cache_size  = cache->wp - cache->rp;

      if (cache_size == 0)
        return PDF_ENINPUT;

      out_size = out->size - out->wp;

      if (MD5_OUTPUT_SIZE > out_size)
        {
          /* We should can return PDF_ENOUTPUT. However the
             output buffer is assumed to be full if we return
             PDF_ENOUTPUT, but we can not write it partially.  */
          return PDF_ERROR;
        }
      else
        {
          pdf_crypt_md_hash (filter_state->md,
                             out->data,
                             out_size,
                             cache->data,
                             cache_size);
        }

      /* Update output buffer */
      cache->rp += cache_size;
      out->wp += MD5_OUTPUT_SIZE;
    }

This code assumes that the output buffer should be at least
MD5_OUTPUT_SIZE in order for the filter to work. Ideally the filters
should work even with --cache=1 in the pdf-filter utility.

What the code should do in that situation is to maintain an internal
cache of size MD5_OUTPUT_SIZE to hold the output of
pdf_crypt_md_hash. Then it should write the hash in the output buffer
as it is available. Finally it should return PDF_EEOF.

So, using pseudocode, the algorithm should be something like:

   if (finish_p == PDF_TRUE)
     {
       if (the temporary output buffer does not contain a hash)
          {
            /* Calculate the hash */

            /* Store the calculated hash in a temporary buffer of size 
               MD5_OUTPUT_SIZE */
          }
        
        /* Write in the output buffer as much as we can */

        if (the temporary output buffer is empty)
         {
            /* We are done */
            return PDF_EEOF;  
         } 
        else
         {
            /* We need more output space */
            return PDF_ENOUTPUT;
         }
     }

I think that the same applies to the AESV2 decoder and encoder:

          if (aux_size > out_size)
            {
              return PDF_ERROR;
            }

What do you think?