[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
|
From: |
Grundmann, Christian |
|
Subject: |
Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc |
|
Date: |
Tue, 17 Nov 2015 14:11:51 +0000 |
Here you go
gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set
pagination off" -ex "thread apply all bt full"
[New LWP 52281]
[New LWP 52288]
[New LWP 52286]
[New LWP 52291]
[New LWP 52292]
[New LWP 52287]
[New LWP 52293]
[New LWP 52290]
[New LWP 56455]
[New LWP 52289]
[New LWP 52282]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine
rhel6.5.0,accel=kvm,us'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4
Thread 11 (Thread 0x7f6d47719700 (LWP 52282)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
No locals.
#1 0x00007f6d57426272 in futex_wait (val=4294967295, ev=0x7f6d57cf0f44
<rcu_call_ready_event>) at util/qemu-thread-posix.c:301
No locals.
#2 qemu_event_wait (address@hidden <rcu_call_ready_event>) at
util/qemu-thread-posix.c:399
value = <optimized out>
#3 0x00007f6d57434526 in call_rcu_thread (opaque=<optimized out>) at
util/rcu.c:233
tries = 0
n = <optimized out>
node = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d47719700) at
pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d47719700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107326789376,
5884348200482620104, 0, 140107326790080, 140107326789376, 140107592666688,
-5804038895876586808, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad
= {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (address@hidden, address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
0x7f6d44f139e0, reg_save_area = 0x7f6d44f139a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570cf000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8a4000) at
/usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8a4000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44f14700) at
pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d44f14700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107284825856,
5884348200482620104, 0, 140107284826560, 140107284825856, 140726431086992,
-5804033392412867896, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad
= {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)):
#0 sem_timedwait () at
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
No locals.
#1 0x00007f6d574260c7 in qemu_sem_timedwait (address@hidden, address@hidden)
at util/qemu-thread-posix.c:254
rc = <optimized out>
ts = {tv_sec = 1447709021, tv_nsec = 21985000}
__func__ = "qemu_sem_timedwait"
#2 0x00007f6d573a98ac in worker_thread (opaque=0x7f6d5a1b91e0) at
thread-pool.c:92
req = <optimized out>
ret = <optimized out>
pool = 0x7f6d5a1b91e0
#3 0x00007f6d55ceadf5 in start_thread (arg=0x7f6ab1dff700) at
pthread_create.c:308
__res = <optimized out>
pd = 0x7f6ab1dff700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140096227505920,
5884348200482620104, 0, 140096227506624, 140096227505920, 26,
-5801757560646548792, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad
= {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#4 0x00007f6d4f8f81ad in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 8 (Thread 0x7f6d44713700 (LWP 52290)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (address@hidden, address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
0x7f6d447129e0, reg_save_area = 0x7f6d447129a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570cc000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8b8000) at
/usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8b8000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44713700) at
pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d44713700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107276433152,
5884348200482620104, 0, 140107276433856, 140107276433152, 140726431086992,
-5804032293438111032, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad
= {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (address@hidden, address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
0x7f6d42f0f9e0, reg_save_area = 0x7f6d42f0f9a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570c3000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8f4000) at
/usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8f4000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d42f10700) at
pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d42f10700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107251255040,
5884348200482620104, 0, 140107251255744, 140107251255040, 140726431086992,
-5804046580109950264, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad
= {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (address@hidden, address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
0x7f6d45f159e0, reg_save_area = 0x7f6d45f159a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570d5000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c878000) at
/usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c878000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45f16700) at
pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d45f16700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107301611264,
5884348200482620104, 0, 140107301611968, 140107301611264, 140726431086992,
-5804035590362381624, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad
= {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 5 (Thread 0x7f6d43711700 (LWP 52292)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (address@hidden, address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
0x7f6d437109e0, reg_save_area = 0x7f6d437109a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570c6000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8e0000) at
/usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8e0000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43711700) at
pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d43711700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107259647744,
5884348200482620104, 0, 140107259648448, 140107259647744, 140726431086992,
-5804047687674641720, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad
= {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (address@hidden, address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
0x7f6d43f119e0, reg_save_area = 0x7f6d43f119a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570c9000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8cc000) at
/usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8cc000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43f12700) at
pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d43f12700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107268040448,
5884348200482620104, 0, 140107268041152, 140107268040448, 140726431086992,
-5804048786649398584, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad
= {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 3 (Thread 0x7f6d46717700 (LWP 52286)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (address@hidden, address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
0x7f6d467169e0, reg_save_area = 0x7f6d467169a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570d8000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c810000) at
/usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c810000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d46717700) at
pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d46717700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107310003968,
5884348200482620104, 0, 140107310004672, 140107310003968, 140726431086992,
-5804036689337138488, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad
= {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 2 (Thread 0x7f6d45715700 (LWP 52288)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (address@hidden, address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
0x7f6d457149e0, reg_save_area = 0x7f6d457149a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (address@hidden) at
/usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570d2000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c890000) at
/usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c890000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45715700) at
pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d45715700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107293218560,
5884348200482620104, 0, 140107293219264, 140107293218560, 140726431086992,
-5804034491387624760, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad
= {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)):
#0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4
No symbol table info available.
#1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=49280) at vl.c:2575
ptr = 0x7f6d59a346a0
#2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0
No symbol table info available.
#3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0
No symbol table info available.
#4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=0x7f6d5de1ff40) at
/usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33
req = <optimized out>
#5 virtio_blk_get_request (s=0x7f6d5de1ff40) at
/usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:192
req = <optimized out>
#6 virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at
/usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604
s = 0x7f6d5de1ff40
__func__ = "virtio_blk_handle_output"
mrb = {reqs = {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7c000, 0x0
<repeats 29 times>}, num_reqs = 3, is_write = true}
#7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=0x7f6d5a1aea00, ret=62,
address@hidden) at iohandler.c:143
revents = 1
#8 0x00007f6d573b7296 in main_loop_wait (nonblocking=<optimized out>) at
main-loop.c:504
ret = 1
timeout = 4294967295
timeout_ns = <optimized out>
#9 0x00007f6d571b6a4e in main_loop () at vl.c:1818
nonblocking = <optimized out>
last_io = 0
#10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at
vl.c:4394
i = <optimized out>
snapshot = <optimized out>
linux_boot = <optimized out>
initrd_filename = <optimized out>
kernel_filename = <optimized out>
kernel_cmdline = <optimized out>
boot_order = 0x7f6d57449a67 "cad"
boot_once = 0x0
cyls = <optimized out>
heads = <optimized out>
secs = <optimized out>
translation = <optimized out>
hda_opts = <optimized out>
opts = <optimized out>
machine_opts = <optimized out>
icount_opts = <optimized out>
olist = <optimized out>
optind = 69
optarg = 0x7f6d5a14b3a0 "rhel6.5.0"
loadvm = <optimized out>
machine_class = <optimized out>
cpu_model = <optimized out>
vga_model = 0x0
qtest_chrdev = <optimized out>
qtest_log = <optimized out>
pid_file = <optimized out>
incoming = <optimized out>
show_vnc_port = <optimized out>
defconfig = <optimized out>
userconfig = 179
log_mask = <optimized out>
log_file = <optimized out>
mem_trace = {malloc = 0x7f6d572a4480 <malloc_and_trace>, realloc =
0x7f6d572a4460 <realloc_and_trace>, free = 0x7f6d572a4450 <free_and_trace>,
calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
trace_events = <optimized out>
trace_file = <optimized out>
maxram_size = <optimized out>
ram_slots = <optimized out>
vmstate_dump_file = <optimized out>
main_loop_err = 0x0
__func__ = "main"
@ Do you think you're only hitting these crashes on VMs that have been paused
because of these space errors?
Will have a look on that
Thx Christian
-----Ursprüngliche Nachricht-----
Von: Dr. David Alan Gilbert [mailto:address@hidden
Gesendet: Dienstag, 17. November 2015 12:36
An: Grundmann, Christian <address@hidden>
Cc: 'address@hidden' <address@hidden>; address@hidden
Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in
libtcmalloc
* Grundmann, Christian (address@hidden) wrote:
> Hi,
>
> @ Can you please use a 'thread apply all bt full' the full gives a little
> more info.
>
> gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set
> pagination off" -ex "thread apply all bt full"
OK, it doesn't relaly give any more without the debuginfo package mentioned
below.
<snip>
> @ Also, if you've not already got it installed can you please install the
> debuginfo package for qemu, it gives a lot more information in backtraces.
> Sorry it's a ovirt-node System where I can't you yum
Ah, although perhaps if you took the core dump, onto another machine with
matching qemu and debuginfo you should be able to get more detail.
> @ Does this part always look the same in your backtraces?
> The most are the same, found one a little bit different :
> Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)):
> #0 0x00007f3785d18353 in
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
> unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info
> available.
> #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from
> /lib64/libtcmalloc.so.4 No symbol table info available.
> #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No
> symbol table info available.
> #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No
> symbol table info available.
> #4 0x00007f37885fec89 in g_slice_free1 () from
> /lib64/libglib-2.0.so.0 No symbol table info available.
> #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table
> info available.
> #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info
> available.
> #7 0x00007f378a398394 in aio_bh_poll () No symbol table info
> available.
> #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table info
> available.
> #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info
> available.
> #10 0x00007f37885e299a in g_main_context_dispatch () from
> /lib64/libglib-2.0.so.0 No symbol table info available.
> #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info
> available.
> #12 0x00007f378a1a5a4e in main ()
> No symbol table info available.
>
OK, that's a bit different but interesting....
> @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> No nothing abnormal
>
> @ 2) Did you hit any IO errors and need to tell the VM to continue after a
> problem?
> Ovirt tells me "no Storage space error". Which is something like the disk is
> growing to fast i think. I use Snapshots so on heavy write the disk has to
> grow a lot.
> Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays
> offline.
OK, that's interesting, because you may be hitting the following bug;
http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html
whose fix coincidentally just got accepted today; it's related to error cases
with error=stop which you are using.
Do you think you're only hitting these crashes on VMs that have been paused
because of these space errors?
> disk emulation and see if the problem goes away - e.g. virtio-scsi would
> be a good one to try.
>
> Ok will try that and report
Thanks,
Dave
>
> Thx Christian
>
>
> -----Ursprüngliche Nachricht-----
> Von: Dr. David Alan Gilbert [mailto:address@hidden
> Gesendet: Dienstag, 17. November 2015 10:59
> An: Grundmann, Christian <address@hidden>
> Cc: 'address@hidden' <address@hidden>;
> address@hidden
> Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in
> libtcmalloc
>
> * Grundmann, Christian (address@hidden) wrote:
> > Hi,
> > Dan sent me over to you,
> > please let me know if i can provide additional informations
>
> Hi Christian,
> Thanks for reporting this,
>
> > Softwareversions:
> > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso
> >
> > qemu-img-ev-2.3.0-29.1.el7.x86_64
> > qemu-kvm-ev-2.3.0-29.1.el7.x86_64
> > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
> > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
> > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
> > kernel-3.10.0-229.14.1.el7.x86_64
> > gperftools-libs-2.4-7.el7.x86_64
> >
> > Commandline:
> > /usr/libexec/qemu-kvm -name myvmname -S -machine
> > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime
> > mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios
> > type=1,manufacturer=oVirt,product=oVirt
> > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-
> > 5A
> > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319
> > -nographic -no-user-config -nodefaults -chardev
> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,se
> > rv er,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc
> > base=2015-11-15T20:04:35,driftfix=slew -global
> > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
> > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> > -device
> > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
> > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
> > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
> > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> > -drive
> > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84
> > -8
> > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffae
> > cf
> > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,
> > fo
> > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,we
> > rr or=stop,rerror=stop,aio=native -device
> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,
> > id
> > =virtio-disk0,bootindex=1 -netdev
> > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device
> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci
> > .0
> > ,addr=0x3 -chardev
> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-
> > 5a 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait
> > -device
> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=cha
> > nn el0,name=com.redhat.rhevm.vdsm -chardev
> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-
> > 5a 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait
> > -device
> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=cha
> > nn
> > el1,name=org.qemu.guest_agent.0 -device
> > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
> >
> > Stack Trace:
> >
> > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set
> > pagination off" -ex "thread apply all bt"
>
> Can you please use a 'thread apply all bt full' the full gives a little
> more info.
> Also, if you've not already got it installed can you please install the
> debuginfo package for qemu, it gives a lot more information in backtraces.
>
> > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)):
> > #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4
> > #1 0x00007fa8b186b489 in malloc_and_trace ()
> > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0
> > #3 0x00007fa8afbd666e in g_slice_alloc () from
> > /lib64/libglib-2.0.so.0
> > #4 0x00007fa8b17cbffd in virtio_blk_handle_output ()
> > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll ()
> > #6 0x00007fa8b197e296 in main_loop_wait ()
> > #7 0x00007fa8b177da4e in main ()
>
> Does this part always look the same in your backtraces?
> The segfault in tc_malloc is probably due to a heap corruption, or double
> free or similar - although it can be a bit tricky to find out what did it,
> since the corruption might have happened a bit before the place it crashed.
>
> Some other ideas:
> 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> 2) Did you hit any IO errors and need to tell the VM to continue after a
> problem?
> 3) If this is pretty repeatable, then it would be interesting to try
> changing to a different
> disk emulation and see if the problem goes away - e.g. virtio-scsi would
> be a good one to try.
>
> Dave
> >
> >
> > Thx Christian
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Dan Kenigsberg [mailto:address@hidden
> > Gesendet: Freitag, 13. November 2015 20:00
> > An: Grundmann, Christian <address@hidden>
> > Cc: 'address@hidden' <address@hidden>
> > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc
> >
> > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote:
> > > Hi,
> > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso"
> > > (is there something better to use?) fort he nodes, and have random
> > > crashes of VMs The dumps are always the Same
> > >
> > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump
> > > [Thread debugging using libthread_db enabled] Using host
> > > libthread_db library "/lib64/libthread_db.so.1".
> > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine
> > > rhel6.5.0,accel=kvm,usb=o'.
> > > Program terminated with signal 11, Segmentation fault.
> > > #0 0x00007f0c559c4353 in
> > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::
> > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> > >
> > >
> > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist
> > > centos7 or 3.6
> >
> > Due to the low-leveled-ness of the problem, I'd guess it's a
> > qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt.
> >
> > Please report the precise version of qemu,kernel,libvirt and
> > gperftools-libs to qemu-devel mailing list and the complete stack trace and
> > qemu command line, if possible.
> >
> --
> Dr. David Alan Gilbert / address@hidden / Manchester, UK
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK
- [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Grundmann, Christian, 2015/11/16
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Dr. David Alan Gilbert, 2015/11/17
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Grundmann, Christian, 2015/11/17
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Dr. David Alan Gilbert, 2015/11/17
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc,
Grundmann, Christian <=
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Grundmann, Christian, 2015/11/17
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Dr. David Alan Gilbert, 2015/11/17
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Grundmann, Christian, 2015/11/19
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Paolo Bonzini, 2015/11/19
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Dr. David Alan Gilbert, 2015/11/20