[Help-gnunet] gnunetd deadlocked

[David Kuehling]

Hi,

my gnunetd daemon seems to have deadlocked after running for some 10
hours.  CPU-load is 0% and 'gnunet-stats' blocks forever

Attaching gdb to the still running process I get the following
information:

info threads:
  13 Thread -1212904528 (LWP 5406)  0xb7ea9189 in __lll_mutex_lock_wait ()
   from /lib/tls/libpthread.so.0
  12 Thread -1217188944 (LWP 5407)  0xb7ea9b3c in __nanosleep_nocancel ()
   from /lib/tls/libpthread.so.0
  11 Thread -1217635408 (LWP 5409)  0xb7e316a7 in select ()
   from /lib/tls/libc.so.6
  10 Thread -1217700944 (LWP 5410)  0xb7e316a7 in select ()
   from /lib/tls/libc.so.6
  9 Thread -1217815632 (LWP 5411)  0xb7ea9189 in __lll_mutex_lock_wait ()
   from /lib/tls/libpthread.so.0
  8 Thread -1219994704 (LWP 5412)  0xb7ea6c01 in pthread_cond_wait@@GLIBC_2.3.2
    () from /lib/tls/libpthread.so.0
  7 Thread -1220133968 (LWP 5413)  0xb7ea9b3c in __nanosleep_nocancel ()
   from /lib/tls/libpthread.so.0
  6 Thread -1220412496 (LWP 5416)  0xb7ea9189 in __lll_mutex_lock_wait ()
   from /lib/tls/libpthread.so.0
  5 Thread -1220478032 (LWP 5417)  0xb7ea6c01 in pthread_cond_wait@@GLIBC_2.3.2
    () from /lib/tls/libpthread.so.0
  4 Thread -1245561936 (LWP 5418)  0xb7ea9189 in __lll_mutex_lock_wait ()
   from /lib/tls/libpthread.so.0
  3 Thread -1237173328 (LWP 5419)  0xb7ea9189 in __lll_mutex_lock_wait ()
   from /lib/tls/libpthread.so.0
  2 Thread -1228784720 (LWP 5420)  0xb7ea9189 in __lll_mutex_lock_wait ()
   from /lib/tls/libpthread.so.0
  1 Thread -1211799872 (LWP 5404)  0xb7ea6c01 in pthread_cond_wait@@GLIBC_2.3.2
    () from /lib/tls/libpthread.so.0

some back-traces:

thread 13
bt
#0  0xb7ea9189 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#1  0xb7ea5f79 in _L_mutex_lock_143 () from /lib/tls/libpthread.so.0
#2  0x0000006c in ?? ()
#3  0x0000fffe in ?? ()
#4  0x00000000 in ?? ()

thread 12
bt
#0  0xb7ea9b3c in __nanosleep_nocancel () from /lib/tls/libpthread.so.0
#1  0xb7d682a0 in PTHREAD_SLEEP (delay=0) at pthread.c:223
#2  0xb7d52423 in cron_main_method (ctx=0x80520a0) at cron.c:593
#3  0xb7ea40bd in start_thread () from /lib/tls/libpthread.so.0
#4  0xb7e389ee in clone () from /lib/tls/libc.so.6

thread 11
bt
#0  0xb7e316a7 in select () from /lib/tls/libc.so.6
#1  0xb7d61dcc in selectThread (ctx=0x805dd48) at select.c:500
#2  0xb7ea40bd in start_thread () from /lib/tls/libpthread.so.0
#3  0xb7e389ee in clone () from /lib/tls/libc.so.6

thread 10
bt
#0  0xb7e316a7 in select () from /lib/tls/libc.so.6
#1  0xb7d61dcc in selectThread (ctx=0x805de10) at select.c:500
#2  0xb7ea40bd in start_thread () from /lib/tls/libpthread.so.0
#3  0xb7e389ee in clone () from /lib/tls/libc.so.6


thread 9 
bt:
#0  0xb7ea9189 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#1  0xb7ea5f79 in _L_mutex_lock_143 () from /lib/tls/libpthread.so.0
#2  0xb7699398 in ?? ()
#3  0xb7d5d851 in xfree_ (ptr=0xfffffffc,
    filename=0xb7ea9b46 "X=\001<snip>"
    function=0xb7d682a0 "\205<snip"
    at xmalloc.c:135
#4  0xb7d67ba6 in MUTEX_LOCK (mutex=0xfffffffc) at mutex.c:134
#5  0xb7fcf981 in forEachConnectedNode (method=0, arg=0x0) at connection.c:3335
#6  0xb76a3c93 in processThread (unused=0x0) at bootstrap.c:146
#7  0xb7ea40bd in start_thread () from /lib/tls/libpthread.so.0
#8  0xb7e389ee in clone () from /lib/tls/libc.so.6

thread 8
bt:
#0  0xb7ea6c01 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/libpthread.so.0
#1  0xb7d688b0 in SEMAPHORE_DOWN (s=0x8060d50, mayblock=1) at semaphore.c:153
#2  0xb76788ed in acquire (key=0x816da68, value=0x816daa8, closure=0x0)
    at prefetch.c:102
#3  0xb766eb13 in iterateHelper (type=<value optimized out>,
    query=<value optimized out>, iter=0xb76788a0 <acquire>, closure=0x0)
    at mysql.c:592
#4  0xb767885d in rcbAcquire (unused=0x0) at prefetch.c:167
#5  0xb7ea40bd in start_thread () from /lib/tls/libpthread.so.0
#6  0xb7e389ee in clone () from /lib/tls/libc.so.6

thread 7
bt:
ep_nocancel () from /lib/tls/libpthread.so.0
(gdb) bt
#0  0xb7ea9b3c in __nanosleep_nocancel () from /lib/tls/libpthread.so.0
#1  0xb7d682a0 in PTHREAD_SLEEP (delay=0) at pthread.c:223
#2  0xb7d52423 in cron_main_method (ctx=0x8095c98) at cron.c:593
#3  0xb7ea40bd in start_thread () from /lib/tls/libpthread.so.0
#4  0xb7e389ee in clone () from /lib/tls/libc.so.6

thread 6
bt:
#0  0xb7ea9189 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#1  0xb7ea5f79 in _L_mutex_lock_143 () from /lib/tls/libpthread.so.0
#2  0x00000724 in ?? ()
#3  0xb7dd0f46 in malloc_usable_size () from /lib/tls/libc.so.6
#4  0xb7d67ba6 in MUTEX_LOCK (mutex=0xfffffffc) at mutex.c:134
#5  0xb7668e82 in select_dht_peer (set=0xb741e948, target=0x83a3c68,
    blocked=0xb741e948, blocked_size=0) at table.c:340
#6  0xb766759e in handlePut (sender=0x0, msg=0x83a3c58) at routing.c:535
#7  0xb7668110 in dht_put (key=0x80b7a90, type=1, size=32772,
    expirationTime=1182603197941, data=0x80b7ae8 "") at routing.c:708
#8  0xb767ef20 in push_callback (key=0x80b7a90, value=0x80b7ad0, closure=0x0)
    at dht_push.c:81
#9  0xb766eb13 in iterateHelper (type=<value optimized out>,
    query=<value optimized out>, iter=0xb767ee50 <push_callback>, closure=0x0)
    at mysql.c:592
#10 0xb766efd4 in iterateNonAnonymous (type=0, on_demand=1,
    iter=0xb767ee50 <push_callback>, closure=0x0) at mysql.c:665
#11 0xb767ee0c in push_thread (cls=0x0) at dht_push.c:96
#12 0xb7ea40bd in start_thread () from /lib/tls/libpthread.so.0
#13 0xb7e389ee in clone () from /lib/tls/libc.so.6

thread 5
bt
#0  0xb7ea6c01 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/libpthread.so.0
#1  0xb7d688b0 in SEMAPHORE_DOWN (s=0x809e320, mayblock=1) at semaphore.c:153
#2  0xb76820da in localGetter (noargs=0x0) at fs.c:993
#3  0xb7ea40bd in start_thread () from /lib/tls/libpthread.so.0
#4  0xb7e389ee in clone () from /lib/tls/libc.so.6

thread 4 
bt
#0  0xb7ea9189 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#1  0xb7ea5f79 in _L_mutex_lock_143 () from /lib/tls/libpthread.so.0
#2  0xb7334080 in ?? ()
#3  0x0000001a in ?? ()
#4  0x0000000a in ?? ()
#5  0x00000598 in ?? ()
#6  0xb5c23298 in ?? ()
#7  0xb7d6b17c in ?? () from /usr/local/lib/libgnunetutil.so.1
#8  0xb7334090 in ?? ()
#9  0x00000000 in ?? ()

thread 3
bt
#0  0xb7ea9189 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#1  0xb7ea5f79 in _L_mutex_lock_143 () from /lib/tls/libpthread.so.0
#2  0x00000048 in ?? ()
#3  0xb7d6b17c in ?? () from /usr/local/lib/libgnunetutil.so.1
#4  0xb4ed9230 in ?? ()
#5  0x00000048 in ?? ()
#6  0xb6423198 in ?? ()
#7  0xb7d6b17c in ?? () from /usr/local/lib/libgnunetutil.so.1
#8  0x00000048 in ?? ()
#9  0x00000048 in ?? ()
#10 0xb6423198 in ?? ()
#11 0xb7d67ba6 in MUTEX_LOCK (mutex=0xfffffffc) at mutex.c:134
#12 0xb7d67ba6 in MUTEX_LOCK (mutex=0xfffffffc) at mutex.c:134
#13 0xb7fd2b2c in unicastCallback (hostId=0x8141f6c, callback=0,
    closure=0xb4ed9230, len=72, importance=0, maxdelay=5000)
    at connection.c:3549
#14 0xb7fd2cd5 in unicast (receiver=0x8141f6c, msg=0xb6423254, importance=0,
    maxdelay=5000) at connection.c:3603
#15 0xb7669690 in considerPeer (sender=0x8141f6c, peer=0x812bf74)
    at table.c:584
#16 0xb766973e in handleDiscovery (sender=0x8141f6c, msg=0x812beec)
    at table.c:645
#17 0xb7fd8229 in injectMessage (sender=0x8141f6c, msg=0x812bea4 "", size=272,
    wasEncrypted=1, session=0x83151b8) at handler.c:433
#18 0xb7fd853a in threadMain (cls=0xbf8bfd48) at handler.c:526
#19 0xb7ea40bd in start_thread () from /lib/tls/libpthread.so.0
#20 0xb7e389ee in clone () from /lib/tls/libc.so.6

thread 2
bt
#0  0xb7ea9189 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#1  0xb7ea5f79 in _L_mutex_lock_143 () from /lib/tls/libpthread.so.0
#2  0x00000000 in ?? ()

thread 1
bt
#0  0xb7ea6c01 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/libpthread.so.0
#1  0xb7d688b0 in SEMAPHORE_DOWN (s=0x804b030, mayblock=1) at semaphore.c:153
#2  0xb7d69069 in GNUNET_SHUTDOWN_WAITFOR () at shutdown.c:59
#3  0x080492f7 in gnunet_main () at gnunetd.c:86
#4  0x080495ea in main (argc=Cannot access memory at address 0x0
) at gnunetd.c:304


I haven't killed the process yet, so just mail me if you need additional
info.  Or send me an id_dsa.pub and I'll try to set up some unprivileged
login access.

best regards,

David
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40