Re: [Qemu-devel] virsh dump (qemu guest memory dump?): KASLR enabled linux guest support

[Daniel P. Berrange]

On Wed, Nov 09, 2016 at 01:20:51PM +0100, Andrew Jones wrote:
> On Wed, Nov 09, 2016 at 11:58:19AM +0000, Daniel P. Berrange wrote:
> > On Wed, Nov 09, 2016 at 12:48:09PM +0100, Andrew Jones wrote:
> > > On Wed, Nov 09, 2016 at 11:37:35AM +0000, Daniel P. Berrange wrote:
> > > > On Wed, Nov 09, 2016 at 12:26:17PM +0100, Laszlo Ersek wrote:
> > > > > On 11/09/16 11:40, Andrew Jones wrote:
> > > > > > On Wed, Nov 09, 2016 at 11:01:46AM +0800, Dave Young wrote:
> > > > > >> Hi,
> > > > > >>
> > > > > >> Latest linux kernel enabled kaslr to randomiz phys/virt memory
> > > > > >> addresses, we had some effort to support kexec/kdump so that crash
> > > > > >> utility can still works in case crashed kernel has kaslr enabled.
> > > > > >>
> > > > > >> But according to Dave Anderson virsh dump does not work, quoted 
> > > > > >> messages
> > > > > >> from Dave below:
> > > > > >>
> > > > > >> """
> > > > > >> with virsh dump, there's no way of even knowing that KASLR
> > > > > >> has randomized the kernel __START_KERNEL_map region, because there 
> > > > > >> is no
> > > > > >> virtual address information -- e.g., like "SYMBOL(_stext)" in the 
> > > > > >> kdump
> > > > > >> vmcoreinfo data to compare against the vmlinux file symbol value.
> > > > > >> Unless virsh dump can export some basic virtual memory data, which
> > > > > >> they say it can't, I don't see how KASLR can ever be supported.
> > > > > >> """
> > > > > >>
> > > > > >> I assume virsh dump is using qemu guest memory dump facility so it
> > > > > >> should be first addressed in qemu. Thus post this query to qemu 
> > > > > >> devel
> > > > > >> list. If this is not correct please let me know.
> > > > > >>
> > > > > >> Could you qemu dump people make it work? Or we can not support 
> > > > > >> virt dump
> > > > > >> as long as KASLR being enabled. Latest Fedora kernel has enabled 
> > > > > >> it in x86_64.
> > > > > >>
> > > > > > 
> > > > > > When the -kernel command line option is used, then it may be 
> > > > > > possible
> > > > > > to extract some information that could be used to supplement the 
> > > > > > memory
> > > > > > dump that dump-guest-memory provides. However, that would be a 
> > > > > > specific
> > > > > > use. In general, QEMU knows nothing about the guest kernel. It 
> > > > > > doesn't
> > > > > > know where it is in the disk image, and it doesn't even know if it's
> > > > > > Linux.
> > > > > > 
> > > > > > Is there anything a guest userspace application could probe from 
> > > > > > e.g.
> > > > > > /proc that would work? If so, then the guest agent could gain a new
> > > > > > feature providing that.
> > > > > 
> > > > > I fully agree. This is exactly what I suggested too, independently, in
> > > > > the downstream thread, before arriving at this upstream thread. Let me
> > > > > quote that email:
> > > > > 
> > > > > On 11/09/16 12:09, Laszlo Ersek wrote:
> > > > > > [...] the dump-guest-memory QEMU command supports an option called
> > > > > > "paging". Here's its documentation, from the "qapi-schema.json" 
> > > > > > source
> > > > > > file:
> > > > > >
> > > > > >> # @paging: if true, do paging to get guest's memory mapping. This 
> > > > > >> allows
> > > > > >> #          using gdb to process the core file.
> > > > > >> #
> > > > > >> #          IMPORTANT: this option can make QEMU allocate several 
> > > > > >> gigabytes
> > > > > >> #                     of RAM. This can happen for a large guest, 
> > > > > >> or a
> > > > > >> #                     malicious guest pretending to be large.
> > > > > >> #
> > > > > >> #          Also, paging=true has the following limitations:
> > > > > >> #
> > > > > >> #             1. The guest may be in a catastrophic state or can 
> > > > > >> have corrupted
> > > > > >> #                memory, which cannot be trusted
> > > > > >> #             2. The guest can be in real-mode even if paging is 
> > > > > >> enabled. For
> > > > > >> #                example, the guest uses ACPI to sleep, and ACPI 
> > > > > >> sleep state
> > > > > >> #                goes in real-mode
> > > > > >> #             3. Currently only supported on i386 and x86_64.
> > > > > >> #
> > > > > >
> > > > > > "virsh dump --memory-only" sets paging=false, for obvious reasons.
> > > > > >
> > > > > > [...] the dump-guest-memory command provides a raw snapshot of the
> > > > > > virtual machine's memory (and of the registers of the VCPUs); it is
> > > > > > not enlightened about the guest.
> > > > > >
> > > > > > If the additional information you are looking for can be retrieved
> > > > > > within the running Linux guest, using an appropriately privieleged
> > > > > > userspace process, then I would recommend considering an extension 
> > > > > > to
> > > > > > the qemu guest agent. The management layer (libvirt, [...]) could
> > > > > > first invoke the guest agent (a process with root privileges running
> > > > > > in the guest) from the host side, through virtio-serial. The new 
> > > > > > guest
> > > > > > agent command would return the information necessary to deal with
> > > > > > KASLR. Then the management layer would initiate the dump like 
> > > > > > always.
> > > > > > Finally, the extra information would be combined with (or placed
> > > > > > beside) the dump file in some way.
> > > > > >
> > > > > > So, this proposal would affect the guest agent and the management
> > > > > > layer (= libvirt).
> > > > > 
> > > > > Given that we already dislike "paging=true", enlightening
> > > > > dump-guest-memory with even more guest-specific insight is the wrong
> > > > > approach, IMO. That kind of knowledge belongs to the guest agent.
> > > > 
> > > > If you're trying to debug a hung/panicked guest, then using a guest
> > > > agent to fetch info is a complete non-starter as it'll be dead.
> > > 
> > > So don't wait. Management software can make this query immediately
> > > after the guest agent goes live. The information needed won't change.
> > 
> > That doesn't help with trying to diagnose a crash during boot up, since
> > the guest agent isn't running till fairly late. I'm also concerned that
> > the QEMU guest agent is likely to be far from widely deployed in guests,
> > so reliance on the guest agent will mean the dump facility is no longer
> > reliably available.
> >
> 
> It'd still be reliably available and useable during early boot, just like
> it is now, for kernels that don't use KASLR. This proposal is only
> attempting to *also* address KASLR kernels, for which there is currently
> no support whatsoever. Call it a best-effort.
> 
> Of course we can get support for [probably] early boot and
> guest-agent-less guests using KASLR too if we introduce a paravirt
> solution, requiring guest kernel and KVM changes. Is it worth it?

There's a standard for persistent storage that is intended to allow
the kernel to dump out data at time of crash:

   https://lwn.net/Articles/434821/

and there's some recent patches to provide a QEMU backend. Could we
leverage that facility to get the data we need from the guest kernel ?

Instead of only using pstore at time of crash, the kernel could see
that its running on KVM, and write out the paging data to pstore. So
when QEMU later generates a core dump, it can grab the corresponding
data from pstore backend ?

Still requires an extra device, to be configured, but at lesat we
would not have to invent yet another paravirt device ourselves, just
use the existing framework.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|