[gnutls-dev] Diffie Hellman interoperability problem with OpenSSL

[Roman Hoog Antink]

Hi

When SSL handshaking with a OpenSSL 0.9.6 server using DH key exchange, gnutls 
version 0.9.7 complains "The Diffie Hellman prime sent by the server is not 
acceptable (not long enough)." and aborts.

After removing the corresponding if-block in lib/auth_dh_common.c line 221, 
the handshake is accomplished as expected and the connection works.

Can someone please explain, what bits = _gnutls_dh_get_prime_bits( session); 
in this context means and why this if-block is important?

How do I have to initialize DH params in my application to avoid this problem?

For easier reading the gnutls code I'm talking about:

    bits = _gnutls_dh_get_prime_bits( session);
    if (bits < 0) {
        gnutls_assert();
        return bits;
    }

    if ( _gnutls_mpi_get_nbits( session->key->client_p) < (size_t)bits) {
        /* the prime used by the peer is not acceptable
         */
        gnutls_assert();
        return GNUTLS_E_DH_PRIME_UNACCEPTABLE;
    }


Roman