Re: [Qemu-discuss] Kernel panic in VMs with largeamountsofmemory(>1TB)

[Burkhard Linke]

Hi,


I've been testing newer releases of qemu to find a stable setup, but to 
no avail yet. Most tests were performed with Ubuntu 16.04 (qemu 2.5, 
qemu 2.8, qemu 2.10.2 from source), but also with the current prerelease 
of Ubuntu 18.04 (using qemu 2.10.1 with patches from former mail).


With the qemu 2.10.1 setup the instance runs into a kernel oops again 
during boot after the initrd is loaded:

....

starting version 229
[   38.465051] random: systemd-udevd: uninitialized urandom read (16 bytes 
read, 10 bits of entropy available)
[   38.467647] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   38.467725] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   38.471182] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   38.471289] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   38.471399] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   38.471505] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   38.471602] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   38.471698] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   38.471807] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   38.574461] usb 1-1: new full-speed USB device number 2 using uhci_hcd
[   38.593696] virtio_net virtio0 ens3: renamed from eth0
[   41.618549] no cont in shutdown!
[   41.619714] floppy0: floppy_shutdown: timeout handler died.
[   41.686587] usb 1-1: New USB device found, idVendor=0627, idProduct=0001
[   41.691627] usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5
[   41.694130] usb 1-1: Product: QEMU USB Tablet
[   41.695894] usb 1-1: Manufacturer: QEMU
[   41.697423] usb 1-1: SerialNumber: 42
[   63.402423] NMI watchdog: BUG: soft lockup - CPU#22 stuck for 23s! 
[systemd-udevd:299]
[   63.408797] Modules linked in: floppy(+)
[   63.409760] CPU: 22 PID: 299 Comm: systemd-udevd Not tainted 
4.4.0-97-generic #120-Ubuntu
[   63.411520] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 
1.10.2-1ubuntu1bcf1 04/01/2014
[   63.413179] task: ffff897e0cdd9c00 ti: ffff897e0ce30000 task.ti: 
ffff897e0ce30000
[   63.414426] NMI watchdog: BUG: soft lockup - CPU#27 stuck for 23s! 
[systemd-udevd:296]
[   63.414431] Modules linked in: floppy(+)
[   63.414434] CPU: 27 PID: 296 Comm: systemd-udevd Not tainted 
4.4.0-97-generic #120-Ubuntu
[   63.414435] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 
1.10.2-1ubuntu1bcf1 04/01/2014
[   63.414436] task: ffff897e0ccaf000 ti: ffff897e0cdd4000 task.ti: 
ffff897e0cdd4000
[   63.414446] RIP: 0010:[<ffffffffc0002fef>]  [<ffffffffc0002fef>] 
set_fdc+0x8f/0x130 [floppy]
[   63.414447] RSP: 0018:ffff897e0cdd7b88  EFLAGS: 00000202
[   63.414448] RAX: 0000000000000000 RBX: 00000000ffffffff RCX: ffffffffc000eac2
[   63.414448] RDX: 00000000000003f4 RSI: 0000000000000000 RDI: 0000000000000000
[   63.414449] RBP: ffff897e0cdd7b90 R08: 0000000000000001 R09: ffffffffffffffff
[   63.414449] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[   63.414450] R13: 0000000000000002 R14: 0000000000000000 R15: 0000000000000001
[   63.414451] FS:  00007ff608ce38c0(0000) GS:ffff8a003f4c0000(0000) 
knlGS:0000000000000000
[   63.414452] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   63.414453] CR2: 00007fa915f35cdc CR3: 0000017e0cdb3000 CR4: 00000000003406e0
[   63.414457] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   63.414457] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   63.414458] Stack:
[   63.414459]  00000000ffffffff ffff897e0cdd7be8 ffffffffc00031e0 
00000000ffffffff
[   63.414460]  0000000000000000 0000000000000000 0000000000000000 
ffff897e0cdd7c10
[   63.414461]  2f7d451a0c407c74 0000000000000000 0000000000000002 
0000000000000002
[   63.414462] Call Trace:
[   63.414467]  [<ffffffffc00031e0>] lock_fdc+0x60/0x130 [floppy]
[   63.414470]  [<ffffffffc00034ab>] user_reset_fdc+0x1b/0x90 [floppy]
[   63.414473]  [<ffffffffc0013b9c>] floppy_module_init+0xa74/0xed8 [floppy]
[   63.414475]  [<ffffffffc0013128>] ? set_cmos+0x68/0x68 [floppy]
[   63.414481]  [<ffffffff8140a61e>] ? kasprintf+0x4e/0x70
[   63.414483]  [<ffffffffc0013128>] ? set_cmos+0x68/0x68 [floppy]
[   63.414489]  [<ffffffff81002123>] do_one_initcall+0xb3/0x200
[   63.414496]  [<ffffffff811ee323>] ? kmem_cache_alloc_trace+0x183/0x1f0
[   63.414504]  [<ffffffff8118e113>] do_init_module+0x5f/0x1cf
[   63.414510]  [<ffffffff8110abdf>] load_module+0x166f/0x1c10
[   63.414512]  [<ffffffff81107180>] ? __symbol_put+0x60/0x60
[   63.414517]  [<ffffffff81216070>] ? kernel_read+0x50/0x80
[   63.414518]  [<ffffffff8110b3c4>] SYSC_finit_module+0xb4/0xe0
[   63.414520]  [<ffffffff8110b40e>] SyS_finit_module+0xe/0x10
[   63.414529]  [<ffffffff818437f2>] entry_SYSCALL_64_fastpath+0x16/0x71
[   63.414542] Code: 04 b6 48 89 f7 48 c1 e0 03 48 8d 88 70 d9 00 c0 0f b6 51 08 83 
e2 03 80 fa 02 74 45 48 8d 04 b6 48 8b 14 c5 70 d9 00 c0 83 c2 04 <ec> 89 c3 ff 
15 58 91 e2 c1 ff 15 52 91 e2 c1 ff 15 4c 91 e2 c1
[   63.463036] RIP: 0010:[<ffffffff81104b88>]  [<ffffffff81104b88>] 
smp_call_function_many+0x1f8/0x260
[   63.464804] RSP: 0018:ffff897e0ce33ad0  EFLAGS: 00000202
[   63.465766] RAX: 0000000000000003 RBX: 0000000000000200 RCX: 0000000000000000
[   63.467080] RDX: ffff887e3fc1b4a0 RSI: 0000000000000200 RDI: ffff897e3fd97d88
[   63.468316] RBP: ffff897e0ce33b08 R08: 0000000000000000 R09: 00000000ffbfffff
[   63.469613] R10: 0000000000000008 R11: ffff897e3fd97d88 R12: ffff897e3fd97d88
[   63.470985] R13: ffff897e3fd97d80 R14: ffffffff81072640 R15: 0000000000000000
[   63.472201] FS:  00007ff608ce38c0(0000) GS:ffff897e3fd80000(0000) 
knlGS:0000000000000000
[   63.473768] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   63.474923] CR2: 00007ff608ce2dcb CR3: 0000017e0ce15000 CR4: 00000000003406e0
[   63.476190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   63.477428] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   63.478693] Stack:
[   63.479274]  0000000000017d40 01ff897e00000001 ffff897e0ce33bf8 
ffffffff81072640
[   63.481030]  0000000000000000 ffff8a003f5d0ee8 ffff8a003f5d0eb8 
ffff897e0ce33b30
[   63.482909]  ffffffff81104ccd ffff897e0ce33bf8 ffff897e0ce33b90 
ffff897e0ce33b50
[   63.484670] Call Trace:
[   63.485278]  [<ffffffff81072640>] ? leave_mm+0xd0/0xd0
[   63.486217]  [<ffffffff81104ccd>] on_each_cpu+0x2d/0x60
[   63.487263]  [<ffffffff81072c2b>] flush_tlb_kernel_range+0x4b/0x80
[   63.488349]  [<ffffffff811cf8ab>] __purge_vmap_area_lazy+0x31b/0x380
[   63.489443]  [<ffffffff811cfa27>] vm_unmap_aliases+0x117/0x140
[   63.490504]  [<ffffffff8106e2fe>] change_page_attr_set_clr+0xee/0x4f0
[   63.491719]  [<ffffffff811416e9>] ? trace_clock_local+0x9/0x10
[   63.492797]  [<ffffffff8106f39f>] set_memory_ro+0x2f/0x40
[   63.493810]  [<ffffffff81106c20>] frob_text.isra.31+0x20/0x30
[   63.498162]  [<ffffffff811081d7>] module_enable_ro+0x27/0x70
[   63.499346]  [<ffffffff8110a232>] load_module+0xcc2/0x1c10
[   63.500363]  [<ffffffff81216070>] ? kernel_read+0x50/0x80
[   63.515889]  [<ffffffff8110b3c4>] SYSC_finit_module+0xb4/0xe0
[   63.516946]  [<ffffffff8110b40e>] SyS_finit_module+0xe/0x10
[   63.517969]  [<ffffffff818437f2>] entry_SYSCALL_64_fastpath+0x16/0x71
[   63.519313] Code: d2 e8 4d ae 30 00 3b 05 bb 56 e3 00 89 c1 0f 8d 99 fe ff ff 48 
98 49 8b 55 00 48 03 14 c5 00 79 f3 81 8b 42 18 a8 01 74 ca f3 90 <8b> 42 18 a8 
01 75 f7 eb bf 0f b6 4d d0 4c 89 fa 4c 89 f6 44 89


I've tried to remove the floppy module only to end up with errors in a 
different module:

Loading, please wait...
starting version 229
[   36.698136] random: systemd-udevd: uninitialized urandom read (16 bytes 
read, 10 bits of entropy available)
[   36.699559] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   36.699644] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   36.703128] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   36.703295] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   36.703389] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   36.703489] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   36.703581] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   36.703682] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   36.703783] random: udevadm: uninitialized urandom read (16 bytes read, 10 
bits of entropy available)
[   36.783239] usb 1-1: new full-speed USB device number 2 using uhci_hcd
[   36.823091] virtio_net virtio0 ens3: renamed from eth0
[   36.866537] AVX2 version of gcm_enc/dec engaged.
[   36.868335] AES CTR mode by8 optimization enabled
[   36.949417] usb 1-1: New USB device found, idVendor=0627, idProduct=0001
[   36.951812] usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5
[   36.954228] usb 1-1: Product: QEMU USB Tablet
[   36.955897] usb 1-1: Manufacturer: QEMU
[   36.957401] usb 1-1: SerialNumber: 42
[   36.983034] hidraw: raw HID events driver (C) Jiri Kosina
[   36.993836] usbcore: registered new interface driver usbhid
[   36.995915] usbhid: USB HID core driver
[   37.007006] input: QEMU QEMU USB Tablet as 
/devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input4
[   37.011320] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v0.01 
Mouse [QEMU QEMU USB Tablet] on usb-0000:00:01.2-1/input0
[   37.131330] tsc: Refined TSC clocksource calibration: 1995.207 MHz
[   37.133681] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 
0x3984fd2cf2b, max_idle_ns: 881590743856 ns
[   63.407171] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! 
[kworker/25:0:138]
[   63.411059] Modules linked in: hid_generic usbhid hid crct10dif_pclmul 
crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul 
glue_helper ablk_helper cryptd psmouse
[   63.414038] CPU: 25 PID: 138 Comm: kworker/25:0 Not tainted 4.4.0-97-generic 
#120-Ubuntu
[   63.415069] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 
1.10.2-1ubuntu1bcf1 04/01/2014
[   63.416304] Workqueue: events_long serio_handle_event
[   63.417023] task: ffff89fe0dbf7000 ti: ffff89fe0d448000 task.ti: 
ffff89fe0d448000
[   63.417974] RIP: 0010:[<ffffffffc0012699>]  [<ffffffffc0012699>] 
vmmouse_detect+0x49/0xb0 [psmouse]
[   63.419301] RSP: 0000:ffff89fe0d44bc58  EFLAGS: 00000246
[   63.420053] RAX: 00000000564d5868 RBX: 0000000000000000 RCX: 000000000000000a
[   63.420942] RDX: 0000000000005658 RSI: 0000000000000001 RDI: ffff89fe0c2f1400
[   63.421814] RBP: ffff89fe0d44bc60 R08: 000000000001a060 R09: ffff89fe0c2f1400
[   63.422688] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000001
[   63.423611] R13: 0000000000000001 R14: 0000000000000014 R15: 0000000000000001
[   63.424472] FS:  0000000000000000(0000) GS:ffff8a003f440000(0000) 
knlGS:0000000000000000
[   63.425567] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   63.426273] CR2: 00007fe83bb1ff4a CR3: 0000000001e0a000 CR4: 00000000003406e0
[   63.427146] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   63.428165] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   63.429029] Stack:
[   63.429379]  ffff89fe0c2f1400 ffff89fe0d44bca0 ffffffffc000163e 
ffff89fe0d44bca0
[   63.430619]  ffff89fe0c2f1400 ffff89fe0c0f2000 ffff89fe0bf40800 
ffff89fe0bf40800
[   63.431995]  0000000000000000 ffff89fe0d44bcd0 ffffffffc0001c84 
ffff89fe0c2f1400
[   63.433231] Call Trace:
[   63.433630]  [<ffffffffc000163e>] psmouse_extensions+0x7e/0x570 [psmouse]
[   63.434501]  [<ffffffffc0001c84>] psmouse_switch_protocol+0x154/0x1d0 
[psmouse]
[   63.435585]  [<ffffffffc00027b0>] psmouse_connect+0x190/0x2f0 [psmouse]
[   63.436356]  [<ffffffff8167c866>] serio_driver_probe+0x36/0x50
[   63.437108]  [<ffffffff81564732>] driver_probe_device+0x222/0x4a0
[   63.437878]  [<ffffffff81564a34>] __driver_attach+0x84/0x90
[   63.438614]  [<ffffffff815649b0>] ? driver_probe_device+0x4a0/0x4a0
[   63.439448]  [<ffffffff8156235c>] bus_for_each_dev+0x6c/0xc0
[   63.440197]  [<ffffffff81563eee>] driver_attach+0x1e/0x20
[   63.440893]  [<ffffffff8167ce11>] serio_handle_event+0x171/0x210
[   63.441633]  [<ffffffff8109a635>] process_one_work+0x165/0x480
[   63.442385]  [<ffffffff8109a99b>] worker_thread+0x4b/0x4c0
[   63.443088]  [<ffffffff8109a950>] ? process_one_work+0x480/0x480
[   63.443942]  [<ffffffff810a0c75>] kthread+0xe5/0x100
[   63.444628]  [<ffffffff810a0b90>] ? kthread_create_on_node+0x1e0/0x1e0
[   63.445439]  [<ffffffff81843b8f>] ret_from_fork+0x3f/0x70
[   63.446151]  [<ffffffff810a0b90>] ? kthread_create_on_node+0x1e0/0x1e0
[   63.447528] Code: 19 48 3d e0 16 de 81 74 11 f6 05 b4 7f 00 00 04 75 51 b8 fa ff 
ff ff 5b 5d c3 ba 58 56 00 00 b9 0a 00 00 00 31 db b8 68 58 4d 56 <ed> 81 fb 68 
58 4d 56 41 89 c0 75 db 83 f8 ff 74 d6 31 c0 45 84
[   70.058017] random: nonblocking pool is initialized

The instance in still active in this state, but the kernel keeps on 
spilling traces to the log.

Both seabios and qemu are patched to support more than 40 bits address 
space width. Any further hint how to resolve this problem is greatly 
appreciated.

Regards,
Burkhard Linke