[Qemu-devel] New Qemu Crash found with evidence of memory corruption

[Alexey Eremenko]

Hi Qemu Developers !

Qumranet's Automated testing reveals, that in some cases Qemu double frees memory and crashes.

Tested with both Qemu-CVS-2007-12-10 and KVM-56 (both Userspace-only and
kernelspace/userspace combo).

Error message:
======================================================

*** glibc detected *** /usr/local/bin/qemu-system-x86_64: double free or
corruption (fasttop): 0x0000000002b6cb10 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3dd0270412]
/lib64/libc.so.6(cfree+0x8c)[0x3dd0273b1c]
/usr/local/bin/qemu-system-x86_64[0x4116c1]
/usr/local/bin/qemu-system-x86_64[0x41403d]
/usr/local/bin/qemu-system-x86_64[0x40889e]
/usr/local/bin/qemu-system-x86_64[0x40db72]
/usr/local/bin/qemu-system-x86_64[0x48cf15]
/usr/local/bin/qemu-system-x86_64[0x48cf9b]
/usr/local/bin/qemu-system-x86_64[0x48d381]
/usr/local/bin/qemu-system-x86_64[0x40dd27]
/usr/local/bin/qemu-system-x86_64[0x40fd03]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3dd021daa4]
/usr/local/bin/qemu-system-x86_64[0x4060b9]
======= Memory map: ========
00400000-0055b000 r-xp 00000000 fd:00 1961296
/usr/local/bin/qemu-system-x86_64
0075b000-0076f000 rw-p 0015b000 fd:00 1961296
/usr/local/bin/qemu-system-x86_64
0076f000-01a3a000 rw-p 0076f000 00:00 0
01a3a000-02a3b000 rwxp 01a3a000 00:00 0
02a3b000-02dcb000 rw-p 02a3b000 00:00 0
[heap]
3dcfe00000-3dcfe1a000 r-xp 00000000 fd:00 1267006
/lib64/ld-2.6.so
3dd0019000-3dd001a000 r--p 00019000 fd:00 1267006
/lib64/ld-2.6.so
3dd001a000-3dd001b000 rw-p 0001a000 fd:00 1267006
/lib64/ld-2.6.so
3dd0200000-3dd0347000 r-xp 00000000 fd:00 1267007
/lib64/libc-2.6.so
3dd0347000-3dd0546000 ---p 00147000 fd:00 1267007
/lib64/libc-2.6.so
3dd0546000-3dd054a000 r--p 00146000 fd:00 1267007
/lib64/libc-2.6.so
3dd054a000-3dd054b000 rw-p 0014a000 fd:00 1267007
/lib64/libc-2.6.so
3dd054b000-3dd0550000 rw-p 3dd054b000 00:00 0
3dd0600000-3dd0602000 r-xp 00000000 fd:00 1267010
/lib64/libdl-2.6.so
3dd0602000-3dd0802000 ---p 00002000 fd:00 1267010
/lib64/libdl-2.6.so
3dd0802000-3dd0803000 r--p 00002000 fd:00 1267010
/lib64/libdl-2.6.so
3dd0803000-3dd0804000 rw-p 00003000 fd:00 1267010
/lib64/libdl-2.6.so
3dd0a00000-3dd0a82000 r-xp 00000000 fd:00 1267009
/lib64/libm-2.6.so
3dd0a82000-3dd0c81000 ---p 00082000 fd:00 1267009
/lib64/libm-2.6.so
3dd0c81000-3dd0c82000 r--p 00081000 fd:00 1267009
/lib64/libm-2.6.so
3dd0c82000-3dd0c83000 rw-p 00082000 fd:00 1267009
/lib64/libm-2.6.so
3dd0e00000-3dd0e14000 r-xp 00000000 fd:00 1267008
/lib64/libz.so.1.2.3
3dd0e14000-3dd1013000 ---p 00014000 fd:00 1267008
/lib64/libz.so.1.2.3
3dd1013000-3dd1014000 rw-p 00013000 fd:00 1267008
/lib64/libz.so.1.2.3
3dd1200000-3dd1215000 r-xp 00000000 fd:00 1267012
/lib64/libpthread-2.6.so
3dd1215000-3dd1414000 ---p 00015000 fd:00 1267012
/lib64/libpthread-2.6.so
3dd1414000-3dd1415000 r--p 00014000 fd:00 1267012
/lib64/libpthread-2.6.so
3dd1415000-3dd1416000 rw-p 00015000 fd:00 1267012
/lib64/libpthread-2.6.so
3dd1416000-3dd141a000 rw-p 3dd1416000 00:00 0
3dd1600000-3dd1704000 r-xp 00000000 fd:00 1953728
/usr/lib64/libX11.so.6.2.0
3dd1704000-3dd1904000 ---p 00104000 fd:00 1953728
/usr/lib64/libX11.so.6.2.0
3dd1904000-3dd190b000 rw-p 00104000 fd:00 1953728
/usr/lib64/libX11.so.6.2.0
3dd1a00000-3dd1a02000 r-xp 00000000 fd:00 1952614
/usr/lib64/libXau.so.6.0.0
3dd1a02000-3dd1c01000 ---p 00002000 fd:00 1952614
/usr/lib64/libXau.so.6.0.0
3dd1c01000-3dd1c02000 rw-p 00001000 fd:00 1952614
/usr/lib64/libXau.so.6.0.0
3dd1e00000-3dd1e05000 r-xp 00000000 fd:00 1953727
/usr/lib64/libXdmcp.so.6.0.0
3dd1e05000-3dd2004000 ---p 00005000 fd:00 1953727
/usr/lib64/libXdmcp.so.6.0.0
3dd2004000-3dd2005000 rw-p 00004000 fd:00 1953727
/usr/lib64/libXdmcp.so.6.0.0
3dd2200000-3dd220d000 r-xp 00000000 fd:00 1267013
/lib64/libgcc_s-4.1.2-20070503.so.1
3dd220d000-3dd240d000 ---p 0000d000 fd:00 1267013
/lib64/libgcc_s-4.1.2-20070503.so.1
3dd240d000-3dd240e000 rw-p 0000d000 fd:00 1267013
/lib64/libgcc_s-4.1.2-20070503.so.1
3dd2600000-3dd2610000 r-xp 00000000 fd:00 1953729
/usr/lib64/libXext.so.6.4.0
3dd2610000-3dd2810000 ---p 00010000 fd:00 1953729
/usr/lib64/libXext.so.6.4.0
3dd2810000-3dd2811000 rw-p 00010000 fd:00 1953729
/usr/lib64/libXext.so.6.4.0
3dd4200000-3dd4209000 r-xp 00000000 fd:00 1953339
/usr/lib64/libXrender.so.1.3.0
3dd4209000-3dd4408000 ---p 00009000 fd:00 1953339
/usr/lib64/libXrender.so.1.3.0
3dd4408000-3dd4409000 rw-p 00008000 fd:00 1953339
/usr/lib64/libXrender.so.1.3.0
3dd4e00000-3dd4e11000 r-xp 00000000 fd:00 1267014
/lib64/libresolv-2.6.so
3dd4e11000-3dd5011000 ---p 00011000 fd:00 1267014
/lib64/libresolv-2.6.so
3dd5011000-3dd5012000 r--p 00011000 fd:00 1267014
/lib64/libresolv-2.6.so
3dd5012000-3dd5013000 rw-p 00012000 fd:00 1267014
/lib64/libresolv-2.6.so
3dd5013000-3dd5015000 rw-p 3dd5013000 00:00 0
3dd5200000-3dd5205000 r-xp 00000000 fd:00 1953732
/usr/lib64/libXfixes.so.3.1.0
3dd5205000-3dd5404000 ---p 00005000 fd:00 1953732
/usr/lib64/libXfixes.so.3.1.0
3dd5404000-3dd5405000 rw-p 00004000 fd:00 1953732

======================================================

GDB shows:

(gdb) c
Continuing.

Program received signal SIGABRT, Aborted.
[Switching to Thread 46912496226896 (LWP 8191)]
0x0000003dd02305b5 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x0000003dd02305b5 in raise () from /lib64/libc.so.6
#1 0x0000003dd0232060 in abort () from /lib64/libc.so.6
#2 0x0000003dd0268d0b in __libc_message () from /lib64/libc.so.6
#3 0x0000003dd0270412 in _int_free () from /lib64/libc.so.6
#4 0x0000003dd0273b1c in free () from /lib64/libc.so.6
#5 0x00000000004116c1 in readline_handle_byte (ch=<value optimized out>)
at /root/Linstall/kvm-56/qemu/readline.c:280
#6 0x000000000041403d in term_read (opaque=<value optimized out>,
buf=0x7fff4089e12d "", size=6) at
/root/Linstall/kvm-56/qemu/monitor.c:2592
#7 0x000000000040889e in tcp_chr_read (opaque=<value optimized out>)
at /root/Linstall/kvm-56/qemu/vl.c:3080
#8 0x000000000040db72 in main_loop_wait (timeout=<value optimized out>)
at /root/Linstall/kvm-56/qemu/vl.c:7178
#9 0x000000000048cf15 in kvm_eat_signals (env=0x2ac75b0, timeout=0)
at /root/Linstall/kvm-56/qemu/qemu-kvm.c:210
#10 0x000000000048cf9b in kvm_main_loop_wait (env=0x2ac75b0, timeout=0)
at /root/Linstall/kvm-56/qemu/qemu-kvm.c:218
#11 0x000000000048d381 in kvm_main_loop_cpu (env=0x2ac75b0)
at /root/Linstall/kvm-56/qemu/qemu-kvm.c:337
#12 0x000000000040dd27 in main_loop () at
/root/Linstall/kvm-56/qemu/vl.c:7238
#13 0x000000000040fd03 in main (argc=<value optimized out>,
argv=<value optimized out>) at /root/Linstall/kvm-56/qemu/vl.c:8978
(gdb)

======================================================
The error seems to be in Qemu's readline.c:

if (idx == TERM_MAX_CMDS) {
/* Need to get one free slot */
free(term_history[0]); <-- Here is the error.
memcpy(term_history, &term_history[1],
&term_history[TERM_MAX_CMDS] - &term_history[1]);
term_history[TERM_MAX_CMDS - 1] = NULL;
idx = TERM_MAX_CMDS - 1;
}


======================================================
Possible workaround:

changing in readline.c from:
#define TERM_MAX_CMDS 64
-to-
#define TERM_MAX_CMDS 4096

======================================================

This bug affects stability of testing, and at least two guest OSes are affected: SUSE Linux 9.1 and OpenBSD 4.1. (Automated setup crashes).

NOTE: I'we been unable to reproduce this crash scenario manually. Perhaps it requires sending a *lot* of commands into Qemu Monitor. Some commands must be illegal, such as "-" sign.

NOTE2: Same bug in KVM bugzilla: http://sourceforge.net/tracker/index.php?func=detail&aid=1851814&group_id=180599&atid=893831

Any ideas?

-Technologov, QA Team Member, Qumranet.