Emacs deadlock: malloc inside signal handler inside malloc ...

[Joe Wells]

Dear Emacs gurus,

I just observed an interesting Emacs deadlock.  It is for a 2-year-old
CVS version, but it seems like it is worth mentioning in case the
design still allows this deadlock.

According to strace, Emacs was stuck inside this call:

  futex(0xb780a320, FUTEX_WAIT, 2, NULL <unfinished ...>

Nothing I did caused any additional system calls to appear on the
strace output, and this system call never returned.

I attached to Emacs with gdb, and got a backtrace (full trace appended
below).  The details I suspect are most relevant are these:

              #0  0xffffe410 in __kernel_vsyscall ()
              #1  0xb77b482e in pthread_setcanceltype () from 
/lib/tls/i686/cmov/libc.so.6
              #2  0x0817839d in emacs_blocked_malloc (size=80) at alloc.c:1220
  malloc ->   #3  0xb77433c5 in malloc () from /lib/tls/i686/cmov/libc.so.6
              ...
              #19 0x080e9ff2 in XTread_socket (sd=0, expected=1, 
hold_quit=0xbfdeb080)
                  at xterm.c:7078
              #20 0x0812326a in read_avail_input (expected=1) at keyboard.c:6651
              #21 0x0812345c in handle_async_input () at keyboard.c:6797
              #22 0x08123503 in input_available_signal (signo=29) at 
keyboard.c:6839
  signal ->   #23 <signal handler called>
  malloc ->   #24 0xb7740688 in malloc_usable_size () from 
/lib/tls/i686/cmov/libc.so.6
              #25 0xb77445e9 in mallopt () from /lib/tls/i686/cmov/libc.so.6
              #26 0x0817884c in allocate_string_data (s=0xab93d28, nchars=739, 
nbytes=1451)
                  at alloc.c:1936
              ...

Notice that malloc is being called inside a signal handler, which has
interrupted malloc!

This happened in my build of an Emacs CVS snapshot from 2005-02-25.
Emacs was built with these commands:

  export CFLAGS='-O0 -g3 -ggdb'
  ./configure --prefix=$HOME/local --enable-debug --disable-nls 
--with-x-toolkit=gtk
  make

I hope this bug can't happen anymore in the latest version!

Joe

P.S.  Additional details below.

----------------------------------------------------------------------
In GNU Emacs 22.0.50.2 (i686-pc-linux-gnu, GTK+ Version 2.8.18)
 of 2006-07-31 on kiwi
Distributor `The X.Org Foundation', version 11.0.70000000
configured using `configure '--prefix=/home/jbw/local' '--enable-debug' 
'--disable-nls' '--with-x-toolkit=gtk' 'CFLAGS=-O0 -g3 -ggdb''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: jbw
  value of $LANG: nil
  locale-coding-system: nil
  default-enable-multibyte-characters: t
----------------------------------------------------------------------
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb77b482e in pthread_setcanceltype () from /lib/tls/i686/cmov/libc.so.6
#2  0x0817839d in emacs_blocked_malloc (size=80) at alloc.c:1220
#3  0xb77433c5 in malloc () from /lib/tls/i686/cmov/libc.so.6
#4  0xb76f3e9a in iconv_close () from /lib/tls/i686/cmov/libc.so.6
#5  0xb76f3ab5 in iconv_open () from /lib/tls/i686/cmov/libc.so.6
#6  0xb7953467 in g_convert_error_quark () from /usr/lib/libglib-2.0.so.0
#7  0xb79534e8 in g_iconv_open () from /usr/lib/libglib-2.0.so.0
#8  0xb79535e3 in g_iconv_close () from /usr/lib/libglib-2.0.so.0
#9  0xb79539a5 in g_convert () from /usr/lib/libglib-2.0.so.0
#10 0xb795412f in g_locale_from_utf8 () from /usr/lib/libglib-2.0.so.0
#11 0xb7c60b5e in gdk_add_client_message_filter ()
   from /usr/lib/libgdk-x11-2.0.so.0
#12 0xb7c615b2 in gdk_x11_register_standard_event_type ()
   from /usr/lib/libgdk-x11-2.0.so.0
#13 0xb7c62c78 in _gdk_events_queue () from /usr/lib/libgdk-x11-2.0.so.0
#14 0xb7c62dc1 in _gdk_events_queue () from /usr/lib/libgdk-x11-2.0.so.0
#15 0xb79678d6 in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0
#16 0xb796a996 in g_main_context_check () from /usr/lib/libglib-2.0.so.0
#17 0xb796ae1e in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0
#18 0xb7dbef64 in gtk_main_iteration () from /usr/lib/libgtk-x11-2.0.so.0
#19 0x080e9ff2 in XTread_socket (sd=0, expected=1, hold_quit=0xbfdeb080)
    at xterm.c:7078
#20 0x0812326a in read_avail_input (expected=1) at keyboard.c:6651
#21 0x0812345c in handle_async_input () at keyboard.c:6797
#22 0x08123503 in input_available_signal (signo=29) at keyboard.c:6839
#23 <signal handler called>
#24 0xb7740688 in malloc_usable_size () from /lib/tls/i686/cmov/libc.so.6
#25 0xb77445e9 in mallopt () from /lib/tls/i686/cmov/libc.so.6
#26 0x0817884c in allocate_string_data (s=0xab93d28, nchars=739, nbytes=1451)
    at alloc.c:1936
#27 0x0817934f in make_uninit_multibyte_string (nchars=739, nbytes=1451)
    at alloc.c:2449
#28 0x0819b6fa in concat (nargs=7, args=0xbfdeb620, target_type=Lisp_String,
    last_special=0) at fns.c:658
#29 0x0819af39 in Fconcat (nargs=7, args=0xbfdeb620) at fns.c:444
#30 0x081cb4af in Fbyte_code (bytestr=147676587, vector=145582964, maxdepth=64)
    at bytecode.c:1079
#31 0x08196655 in funcall_lambda (fun=146592524, nargs=1,
    arg_vector=0xbfdeb814) at eval.c:2967
#32 0x08196089 in Ffuncall (nargs=2, args=0xbfdeb810) at eval.c:2828
#33 0x081ca5f6 in Fbyte_code (bytestr=147676315, vector=147879068, maxdepth=16)
    at bytecode.c:686
#34 0x08196655 in funcall_lambda (fun=147879348, nargs=1,
    arg_vector=0xbfdeb9f4) at eval.c:2967
#35 0x08196089 in Ffuncall (nargs=2, args=0xbfdeb9f0) at eval.c:2828
#36 0x081ca5f6 in Fbyte_code (bytestr=147677067, vector=141037340, maxdepth=48)
    at bytecode.c:686
#37 0x08196655 in funcall_lambda (fun=147825172, nargs=0,
    arg_vector=0xbfdebbe4) at eval.c:2967
#38 0x08196089 in Ffuncall (nargs=1, args=0xbfdebbe0) at eval.c:2828
#39 0x081ca5f6 in Fbyte_code (bytestr=147644587, vector=141234812, maxdepth=32)
    at bytecode.c:686
#40 0x08196655 in funcall_lambda (fun=147057724, nargs=0,
    arg_vector=0xbfdebe38) at eval.c:2967
#41 0x08196089 in Ffuncall (nargs=1, args=0xbfdebe34) at eval.c:2828
#42 0x08195907 in run_hook_with_args (nargs=1, args=0xbfdebe34,
    cond=to_completion) at eval.c:2448
#43 0x081956e9 in Frun_hooks (nargs=1, args=0xbfdebf14) at eval.c:2311
#44 0x08195d9b in Ffuncall (nargs=2, args=0xbfdebf10) at eval.c:2761
#45 0x08195ab3 in call1 (fn=137826217, arg1=137793417) at eval.c:2574
#46 0x0811c831 in safe_run_hooks_1 (hook=-1075921040) at keyboard.c:2044
#47 0x08193caa in internal_condition_case (bfun=0x811c815 <safe_run_hooks_1>,
    handlers=137740425, hfun=0x811c833 <safe_run_hooks_error>) at eval.c:1385
#48 0x0811c8cb in safe_run_hooks (hook=137793417) at keyboard.c:2072
#49 0x0811bb11 in command_loop_1 () at keyboard.c:1811
#50 0x08193caa in internal_condition_case (bfun=0x811a6c6 <command_loop_1>,
    handlers=137801377, hfun=0x811a200 <cmd_error>) at eval.c:1385
#51 0x0811a547 in command_loop_2 () at keyboard.c:1319
#52 0x0819376f in internal_catch (tag=137795377,
    func=0x811a523 <command_loop_2>, arg=137740377) at eval.c:1144
#53 0x0811a4fc in command_loop () at keyboard.c:1298
#54 0x08119f82 in recursive_edit_1 () at keyboard.c:991
#55 0x0811a0c7 in Frecursive_edit () at keyboard.c:1052
#56 0x08118a0c in main (argc=1, argv=0xbfdec854) at emacs.c:1766