[Help-gnutls] PKCS#8 incompatibility? between OpenSSL and GnuTLS

[Kukosa, Tomas]

Hi,
 
I have recived PKCS#12 file created with OpenSSL 0.9.7e which I can not
read in GnuTLS 2.7.12 but I still can read it in any OpenSSL.

When I extracted essential problem which seems to be
decryption/encryption of PKCS#8 I came to the following result:

There are fixed RSA private key key01.pem and password "123456".

Let's encrypt it with OpenSSL (tested with 0.9.7e and 0.9.8k)
>openssl pkcs8 -topk8 -in key01.pem -passout pass:123456 -out
.\data\x_NNNN.pem -v1 PBE-SHA1-3DES

Then let's decrypt it with GnuTLS (tested with 2.7.12)
>certtool -k --password 123456 --infile .\data\x_NNNN.pem --outfile
.\data\y_NNN.pem

It can be usuallay decrypted without any problem. But if you try it more
times (tested 90000 times with OpenSSL 0.9.7e and 9000 times with
0.9.8k) it can not be decrypted in about 0,8% of all cases.
It fails during decryption:
>certtool.exe -d 9 -k --password 123456 --infile .\data\x_9607.pem
Setting log level to 9
|<2>| ASSERT: ../../../src/gnutls-2.8.0/lib/x509_b64.c:452
|<2>| Could not find '-----BEGIN RSA PRIVATE KEY'
|<2>| ASSERT: ../../../src/gnutls-2.8.0/lib/x509_b64.c:452
|<2>| Could not find '-----BEGIN DSA PRIVATE KEY'
|<2>| ASSERT: ../../../../src/gnutls-2.8.0/lib/x509/privkey.c:373
|<2>| ASSERT: ../../../src/gnutls-2.8.0/lib/x509_b64.c:452
|<2>| Could not find '-----BEGIN PRIVATE KEY'
|<2>| ASSERT: ../../../../src/gnutls-2.8.0/lib/x509/privkey_pkcs8.c:972
|<2>| ASSERT: ../../../../src/gnutls-2.8.0/lib/x509/privkey_pkcs8.c:1118
|<2>| ASSERT: ../../../src/gnutls-2.8.0/lib/x509_b64.c:452
|<2>| Could not find '-----BEGIN PRIVATE KEY'
|<9>| salt.size: 8
|<9>| iterationCount: 2048
|<2>| ASSERT: ../../../../src/gnutls-2.8.0/lib/x509/privkey_pkcs8.c:972
|<2>| ASSERT: ../../../../src/gnutls-2.8.0/lib/x509/privkey_pkcs8.c:836
|<2>| ASSERT: ../../../../src/gnutls-2.8.0/lib/x509/privkey_pkcs8.c:1118
certtool.exe: import error: Decryption has failed.

Those erroneous keys still can be decrypted with OpenSSL.

The attached file contains all test scripts and few encrypted PKCS#8
files.
Files x_9607.pem, x_9671.pem, x_9926.pem, x_9931.txt contain erroneous
keys.

How to find whether it is bug in OpenSSL or GnuTLS?

BTW 0,8% is near to 1/128 or to 1/120 but it could be just random :-)

Any ideas are welcome!

Best regards,
  Tomas

test-with-openssl-0.9.8k.zip
Description: test-with-openssl-0.9.8k.zip