Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[Josh Triplett]

On Thu, Sep 03, 2015 at 11:16:40AM +0200, Laszlo Ersek wrote:
> Hi Josh,
> 
> I just found the LWN.net article with the title in the subject:
> 
>   http://lwn.net/SubscriberLink/655992/04701db2bbb7e716/
> 
> ("Thank you LWN", of course. :))
> 
> This got me kind of excited. In your presentation, according to the
> article, you were using KVM and OVMF to showcase the ACPI features of
> "BITS". Further, in
> 
>   http://biosbits.org/scripting/

(Note: that page is badly out of date, by a couple of years, and we need
to drop it in favor of the various more recent presentations.  All of
that functionality still exists, but we have an interactive Python REPL
now, and the ACPI methods have moved to the "acpi" module and become
easier to use.)

> you write, "You can also find arbitrary ACPI objects by path, and
> evaluate arbitrary ACPI methods, with the return value decoded into a
> Python object".

Right.  In particular, you can call acpi.get_objpaths to list the ACPI
tree, and acpi.evaluate to evaluate an arbitrary method.

> I'm excited because for virtualization we might have the reverse use
> case: use BITS to test / debug / develop the ACPI generator of QEMU. :)

Awesome.  Unit testing and CI for QEMU sounds like an ideal use case for
BITS.

> As you may now, QEMU generates almost all of its ACPI tables
> dynamically, providing the guest with an "ACPI payload" that describes
> the actual virtual hardware as accurately as possible.

Many BIOSes do that, since the hardware configuration may vary at
runtime.

> Then this payload is passed to the guest firmware (SeaBIOS or OVMF) over
> "fw_cfg" (which is a simple protocol, comprising, at this point, one
> selector and one data register, which are IO ports or MMIO locations --
> see "docs/specs/fw_cfg.txt" in QEMU and
> "Documentation/devicetree/bindings/arm/fw-cfg.txt" in the kernel).

Interesting; I hadn't seen that protocol before.

Do you virtualize those I/O ports by CPU, to make them thread-safe, or
does the last address written to 0x510 get saved system-wide, making it
unsafe for concurrent access?

> The guest firmware downloads the "ACPI payload" over fw_cfg, does some
> generic processing on them (we call this the "ACPI linker/loader
> script"), and ultimately installs the ACPI tables in a way that is
> appropriate for the firmware in question.
> 
> To be more concrete, the tables you investigated in your presentation
> with BITS were generated (very roughly) by the following files in QEMU:
> 
>   hw/acpi/aml-build.c
>   hw/acpi/bios-linker-loader.c
>   hw/acpi/...
>   hw/i386/acpi-build.c
> 
> and processed and installed by the following source file in edk2:
> 
>   OvmfPkg/AcpiPlatformDxe/QemuFwCfgAcpi.c
> 
> With this background, you can probably see where I'm going with this. It
> is not really easy to *test* the AML methods that QEMU generates
> (piecing them together, with helper functions, from AML primitives),
> without dedicated guest kernel drivers. I think the only method that I
> know of is the following:
> 
> - in the Linux guest, dump the ACPI tables with acpidump, or save them
> from sysfs directly (/sys/firmware/acpi/tables)
> - pass the DSDT and the SSDTs (and any data tables referenced by them?)
> to AcpiExec from the ACPICA suite
> - work with AcpiExec
> 
> But, for simple testing, can we maybe run your tool within the guest,
> before the runtime OS boots?

Yes, absolutely.  We have a batch-mode testing mechanism based on a
config file; you'd probably want to make use of that.  With some
extensions, it could dump results either to an emulated serial port or
some other interface that you can read from outside qemu.  We also need
to work on making the results more machine-parseable for automation.

> Although AcpiDump is being ported to edk2:
> 
>   http://thread.gmane.org/gmane.comp.bios.edk2.devel/1109
> 
> I'm unsure if it will be able to evaluate / execute AML methods. (If it
> parallels the "acpidump" utility we have on GNU/Linux distros, it won't.)

Probably not.  And I haven't heard of any plans to port acpiexec to
EDK2.

> Thus it would be awesome if we had some AcpiExec-like functionality
> early on in the guest (for example in the form of a UEFI Shell
> application, or as a python tool that runs within the edk2 Python port,
> or even in grub).
> 
> For example, assume your runtime guest OS is Windows (with its picky,
> closed-source ACPI interpreter); you make a change in QEMU's ACPI
> generator, rebuild QEMU, reboot the guest, drop to the UEFI shell to
> verify the change "by eye", exit the shell, and *then* continue booting
> Windows. (Which will hopefully not BSOD at that point, after the
> verification with BITS / AcpiExec etc.)
> 
> So, I have three questions:
> 
> (1) What is the relationship between the ACPI facility of BITS, and ACPICA?

BITS links in ACPICA and uses it to evaluate ACPI.  We pull in ACPICA as
a git submodule and build it as part of BITS.  acpi.evaluate uses the
ACPICA interpreter.

> (2) Is there a bit more comprehensive documentation about the ACPI
> module of BITS? AcpiExec and the ACPICA Debugger have quite indulged me
> with their incredible documentation (documents/acpica-reference.pdf). It
> would be great if BITS' ACPI module had a list of commands, to see what
> is there to play with.

We don't come close to the level of documentation for ACPICA, but we do
have pydoc documentation for the modules in BITS, including acpi.  You
can run help("acpi") within BITS, or read acpi.py.  We've tried to make
sure all of the methods considered "API" have docstrings.

> (3) I should mention that QEMU generates ACPI also for arm/aarch64
> virtual machines ("hw/arm/virt-acpi-build.c" in QEMU), and the same edk2
> module as noted above (built for arm/aarch64) does the guest side
> processing. Do you think it's possible to use BITS in arm/aarch64 VMs?

Some folks from Linaro started looking into a BITS arm64 port.

> ... I apologize if tools / documentation already exist for this kind of
> development work; everyone please educate me then. I hope my questions
> make at least some sense; I realize this email isn't well organized.

Makes perfect sense, and thanks for your mail!  I love the idea of using
BITS to test qemu's own ACPI.

- Josh Triplett