GNUTLS handshake errors and memory leaks (ECDHE related?)

I have added SSL support to my web server (nxweb) using GNUTLS. The server has asynchronous architecture (based on epoll), where 4 threads run almost independently accepting and servicing client http connections. Therefore I am using GNUTLS in non-blocking mode.

Platform: Ubuntu 11.10, x64, 4-core CPU.

Linked with latest GNUTLS 3.0.11.

Most of the time everything works fine. Benchmarks show significant speed advantage over nginx (which is build on OpenSSL). But sometimes there are handshake errors and also memory leaks, which might be bound to those errors.

I am attaching full valgrind log. Here are some excerpts:

% valgrind --leak-check=full --show-reachable=yes dist/Debug/GNU-Linux-x86/nxweb

2012-01-10 13:46:51 [28704:0x71b9700]: gnutls handshake error -24 fatal=1

2012-01-10 13:47:25 [28704:0x71b9700]: gnutls handshake error -110 fatal=1

2012-01-10 13:47:25 [28704:0x71b9700]: gnutls handshake error -10 fatal=1

==28704== 47,952 bytes in 999 blocks are definitely lost in loss record 11 of 11

==28704== at 0x4C28F9F: malloc (vg_replace_malloc.c:236)

==28704== by 0x5F28908: __gmp_default_allocate (in /usr/lib/libgmp.so.10.0.1)

==28704== by 0x5F3CF39: __gmpz_mul (in /usr/lib/libgmp.so.10.0.1)

==28704== by 0x50ED444: ecc_projective_check_point (ecc_projective_check_point.c:60)

==28704== by 0x50E692A: _wrap_nettle_pk_derive (pk.c:144)

==28704== by 0x50CD4A2: calc_ecdh_key (ecdh_common.c:64)

==28704== by 0x50CD569: _gnutls_proc_ecdh_common_client_kx (ecdh_common.c:124)

==28704== by 0x50615E6: _gnutls_recv_client_kx_message (gnutls_kx.c:549)

==28704== by 0x505ED25: _gnutls_handshake_server (gnutls_handshake.c:2801)

==28704== by 0x505F622: gnutls_handshake (gnutls_handshake.c:2368)

==28704== by 0x4088FD: do_handshake (nxd_ssl_socket.c:158)

==28704== by 0x408A62: handshake_stub_is_do_write (nxd_ssl_socket.c:188)

There are also other leaks related to gnutls_dh_params_generate2 and some error messages related to gnutls_certificate_set_x509_key_file but those are of less importance to me as they happen only once on server startup.

The log has been collected while running apache-bench (ab) on the client-side. Here is the log:

% ab -c 100 -n 1000 'https://test.domain.ru:8056/benchmark'

This is ApacheBench, Version 2.3 <$Revision: 655654 $>

Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking test.domain.ru (be patient)

Completed 100 requests

Completed 200 requests

Completed 300 requests

Completed 400 requests

Completed 500 requests

Completed 600 requests

Completed 700 requests

SSL handshake failed (5).

Completed 800 requests

Completed 900 requests

Completed 1000 requests

Finished 1000 requests

Server Software: nxweb/3.0.0-dev

Server Hostname: test.domain.ru

Server Port: 8056

SSL/TLS Protocol: TLSv1/SSLv3,ECDHE-RSA-AES256-SHA,2048,256

Document Path: /benchmark

Document Length: 20 bytes

Concurrency Level: 100

Time taken for tests: 159.929 seconds

Complete requests: 1000

Failed requests: 3

(Connect: 0, Receive: 0, Length: 3, Exceptions: 0)

Write errors: 0

Total transferred: 164505 bytes

HTML transferred: 19940 bytes

Requests per second: 6.25 [#/sec] (mean)

Time per request: 15992.885 [ms] (mean)

Time per request: 159.929 [ms] (mean, across all concurrent requests)

Transfer rate: 1.00 [Kbytes/sec] received

Connection Times (ms)

min mean[+/-sd] median max

Connect: 0 13464 1915.7 13581 16479

Processing: 115 2526 1904.2 2416 15897

Waiting: 115 2487 1758.1 2411 15785

Total: 15870 15989 212.0 15912 16645

The problems might be related to ECDHE algorithm. There were no handshake errors when 'PERFORMANCE:%SERVER_PRECEDENCE' priority string was used. There were no handshake errors when using GNUTLS 2.10.5 (from Ubuntu repo) -- I switched to 3.0 version as it supports significantly faster ECDHE algorithm, to be on par with nginx/openssl.

And there seemed to be no such leaks when there were no handshake erros.

And, yes, I do call gnutls_deinit(session) when closing every connection regardless of errors on it. I am ready to provide full source code (I plan to publish it soon as it is open source).

From:	Yaroslav
Subject:	GNUTLS handshake errors and memory leaks (ECDHE related?)
Date:	Tue, 10 Jan 2012 14:44:30 +0400