Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[Moore, Robert]

> -----Original Message-----
> From: Laszlo Ersek [mailto:address@hidden
> Sent: Thursday, September 03, 2015 8:54 AM
> To: Josh Triplett
> Cc: edk2-devel-01; qemu devel list; Michael Tsirkin; Igor Mammedov; Marcel
> Apfelbaum; Paolo Bonzini; Shannon Zhao; Moore, Robert; Smith, Jonathan D;
> Jake Edge
> Subject: Re: "Using Python to investigate EFI and ACPI"
> 
> On 09/03/15 16:50, Josh Triplett wrote:
> > On Thu, Sep 03, 2015 at 11:16:40AM +0200, Laszlo Ersek wrote:
> >> Hi Josh,
> >>
> >> I just found the LWN.net article with the title in the subject:
> >>
> >>   http://lwn.net/SubscriberLink/655992/04701db2bbb7e716/
> >>
> >> ("Thank you LWN", of course. :))
> >>
> >> This got me kind of excited. In your presentation, according to the
> >> article, you were using KVM and OVMF to showcase the ACPI features of
> >> "BITS". Further, in
> >>
> >>   http://biosbits.org/scripting/
> >
> > (Note: that page is badly out of date, by a couple of years, and we
> > need to drop it in favor of the various more recent presentations.
> > All of that functionality still exists, but we have an interactive
> > Python REPL now, and the ACPI methods have moved to the "acpi" module
> > and become easier to use.)
> >
> >> you write, "You can also find arbitrary ACPI objects by path, and
> >> evaluate arbitrary ACPI methods, with the return value decoded into a
> >> Python object".
> >
> > Right.  In particular, you can call acpi.get_objpaths to list the ACPI
> > tree, and acpi.evaluate to evaluate an arbitrary method.
> >
> >> I'm excited because for virtualization we might have the reverse use
> >> case: use BITS to test / debug / develop the ACPI generator of QEMU.
> >> :)
> >
> > Awesome.  Unit testing and CI for QEMU sounds like an ideal use case
> > for BITS.
> >
> >> As you may now, QEMU generates almost all of its ACPI tables
> >> dynamically, providing the guest with an "ACPI payload" that
> >> describes the actual virtual hardware as accurately as possible.
> >
> > Many BIOSes do that, since the hardware configuration may vary at
> > runtime.
> >
> >> Then this payload is passed to the guest firmware (SeaBIOS or OVMF)
> >> over "fw_cfg" (which is a simple protocol, comprising, at this point,
> >> one selector and one data register, which are IO ports or MMIO
> >> locations -- see "docs/specs/fw_cfg.txt" in QEMU and
> >> "Documentation/devicetree/bindings/arm/fw-cfg.txt" in the kernel).
> >
> > Interesting; I hadn't seen that protocol before.
> >
> > Do you virtualize those I/O ports by CPU, to make them thread-safe, or
> > does the last address written to 0x510 get saved system-wide, making
> > it unsafe for concurrent access?
> 
> I think fw_cfg is not meant to be accessed by several CPUs concurrently.
> The protocol is stateful (selected key, offset within blob associated with
> selected key, etc), and "accessing CPU" is not part of that state.
> 
> >
> >> The guest firmware downloads the "ACPI payload" over fw_cfg, does
> >> some generic processing on them (we call this the "ACPI linker/loader
> >> script"), and ultimately installs the ACPI tables in a way that is
> >> appropriate for the firmware in question.
> >>
> >> To be more concrete, the tables you investigated in your presentation
> >> with BITS were generated (very roughly) by the following files in QEMU:
> >>
> >>   hw/acpi/aml-build.c
> >>   hw/acpi/bios-linker-loader.c
> >>   hw/acpi/...
> >>   hw/i386/acpi-build.c
> >>
> >> and processed and installed by the following source file in edk2:
> >>
> >>   OvmfPkg/AcpiPlatformDxe/QemuFwCfgAcpi.c
> >>
> >> With this background, you can probably see where I'm going with this.
> >> It is not really easy to *test* the AML methods that QEMU generates
> >> (piecing them together, with helper functions, from AML primitives),
> >> without dedicated guest kernel drivers. I think the only method that
> >> I know of is the following:
> >>
> >> - in the Linux guest, dump the ACPI tables with acpidump, or save
> >> them from sysfs directly (/sys/firmware/acpi/tables)
> >> - pass the DSDT and the SSDTs (and any data tables referenced by
> >> them?) to AcpiExec from the ACPICA suite
> >> - work with AcpiExec
> >>
> >> But, for simple testing, can we maybe run your tool within the guest,
> >> before the runtime OS boots?
> >
> > Yes, absolutely.  We have a batch-mode testing mechanism based on a
> > config file; you'd probably want to make use of that.  With some
> > extensions, it could dump results either to an emulated serial port or
> > some other interface that you can read from outside qemu.  We also
> > need to work on making the results more machine-parseable for
> automation.
> 
> While I certainly don't discount automation, my primary use case is
> interactive development / testing. :) (Although, I can see myself canning
> some commands in a script or config file, and invoking *that*
> interactively. Which was your point, probably.)
> 
> >
> >> Although AcpiDump is being ported to edk2:
> >>
> >>   http://thread.gmane.org/gmane.comp.bios.edk2.devel/1109
> >>
> >> I'm unsure if it will be able to evaluate / execute AML methods. (If
> >> it parallels the "acpidump" utility we have on GNU/Linux distros, it
> >> won't.)
> >
> > Probably not.  And I haven't heard of any plans to port acpiexec to
> > EDK2.
> >
> >> Thus it would be awesome if we had some AcpiExec-like functionality
> >> early on in the guest (for example in the form of a UEFI Shell
> >> application, or as a python tool that runs within the edk2 Python
> >> port, or even in grub).
> >>
> >> For example, assume your runtime guest OS is Windows (with its picky,
> >> closed-source ACPI interpreter); you make a change in QEMU's ACPI
> >> generator, rebuild QEMU, reboot the guest, drop to the UEFI shell to
> >> verify the change "by eye", exit the shell, and *then* continue
> >> booting Windows. (Which will hopefully not BSOD at that point, after
> >> the verification with BITS / AcpiExec etc.)
> >>
> >> So, I have three questions:
> >>
> >> (1) What is the relationship between the ACPI facility of BITS, and
> ACPICA?
> >
> > BITS links in ACPICA and uses it to evaluate ACPI.  We pull in ACPICA
> > as a git submodule and build it as part of BITS.  acpi.evaluate uses
> > the ACPICA interpreter.
> 
> Awesome! :)
> 
> Another question: when you execute an AML method that does, say, IO port
> access, does the AML interpreter of ACPICA actually *perform* that IO port
> access? Because, the one that is embedded in Linux obviously does, and the
> one that is embedded in the userspace ACPICA command line utility
> "acpiexec" obviously doesn't.

We have recently integrated the ACPICA AML debugger with the Linux kernel,
and we have a userspace utility that allows communication with the debugger.
This allows the user to enter debugger commands such as "execute control 
method",
and the commands are carried out by the kernel code -- thus talking to the
actual hardware.

Bob

> 
> I assume (and very much hope) that the IO port access *is* performed from
> BITS, simply because you developed it for physical machines, and it
> wouldn't make much sense to avoid actual hardware access that was
> implemented by the BIOS vendor for that platform.
> 
> If that is the case, then this tool could become the killer ACPI tester
> for QEMU developers -- the hardware accesses in the AML methods generated
> by QEMU would actually poke QEMU devices! (Unlike the userspace "acpiexec"
> utility.) It would completely detach Linux guest driver development from
> host side / firmware development. \o/
> 
> >
> >> (2) Is there a bit more comprehensive documentation about the ACPI
> >> module of BITS? AcpiExec and the ACPICA Debugger have quite indulged
> >> me with their incredible documentation
> >> (documents/acpica-reference.pdf). It would be great if BITS' ACPI
> >> module had a list of commands, to see what is there to play with.
> >
> > We don't come close to the level of documentation for ACPICA, but we
> > do have pydoc documentation for the modules in BITS, including acpi.
> > You can run help("acpi") within BITS, or read acpi.py.  We've tried to
> > make sure all of the methods considered "API" have docstrings.
> 
> I'll have to digest this some, and play with it.
> 
> >> (3) I should mention that QEMU generates ACPI also for arm/aarch64
> >> virtual machines ("hw/arm/virt-acpi-build.c" in QEMU), and the same
> >> edk2 module as noted above (built for arm/aarch64) does the guest
> >> side processing. Do you think it's possible to use BITS in arm/aarch64
> VMs?
> >
> > Some folks from Linaro started looking into a BITS arm64 port.
> 
> Great!
> 
> >> ... I apologize if tools / documentation already exist for this kind
> >> of development work; everyone please educate me then. I hope my
> >> questions make at least some sense; I realize this email isn't well
> organized.
> >
> > Makes perfect sense, and thanks for your mail!  I love the idea of
> > using BITS to test qemu's own ACPI.
> 
> Thank you very much! :)
> 
> (I must say, I have found the LWN article at just the right time. I intend
> to start implementing a VMGenID device for QEMU, and it's all ACPI based.
> Here's our design for that:
> <http://thread.gmane.org/gmane.comp.emulators.qemu/357940>. I've been
> already dreading the need for a Linux guest driver, in order to white-box
> test the device & the ACPI stuff from the guest side. :))
> 
> Cheers!
> Laszlo