On Tue, Dec 18, 2012 at 10:41 AM, Vasilis Liaskovitis
<address@hidden> wrote:
This is v4 of the ACPI memory hotplug functionality. Only x86_64 target is
supported (both i440fx and q35). There are still several issues, but it's
been a while since v3 and I wanted to get some more feedback on the current
state of the patchseries.
We are working in memory hotplug functionality on pSeries machine. I'm wondering whether and how we can better integrate things. Do you think the DIMM abstraction is generic enough to be used in other machine types?
Overview:
Dimm device layout is modeled with a normal qemu device:
"-device dimm,id=name,size=sz,node=pxm,populated=on|off,bus=membus.0"
How does this will handle the no-hotplugable memory for example the memory passed in '-m' parameter?
The starting physical address for all dimms is calculated from top of memory,
during memory controller init, skipping the pci hole at [PCI_HOLE_START, 4G).
e.g.
"-device dimm,id=dimm0,size=512M,node=0,populated=off,bus=membus.0"
will define a 512M memory dimm belonging to numa node 0, on bus membus.0.
Because dimm layout needs to be configured on machine-boot, all dimm devices
need to be specified on startup command line (either with populated=on or with
populated=off). The dimm information is stored in dimm configuration structures.
After machine startup, dimms are hot-added or removed with normal device_add
and device_del operations e.g.:
Hot-add syntax: "device_add dimm,id=mydimm0,bus=membus.0"
Hot-remove syntax: "device_del dimm,id=mydimm0"
Changes v3->v4
- Dimms added with normal -device argument (extra -dimm arg dropped).
- multiple memory buses can be registered. Memory buses of the real hw/chipset
or a paravirtual memory bus can be added.
- acpi implementation uses memory API instead of old ioports.
- Support for q35/ich9 added (still buggy, see patch 12/31).
- piix4/i440fx initialization code has been refactored to resemble q35. This
will allow memory map initialization at chipset qdev init time for both
machines, as well as more similar code.
- Hot-remove functionality has been moved to separate patches. Hot-remove no
longer frees memory but unmaps the dimm/qdev device from the guest's view.
Freeing the memory should happen when the last user unrefs/unmaps the memory,
see also (work in progress):
https://lists.gnu.org/archive/html/qemu-devel/2012-11/msg00728.html
https://lists.gnu.org/archive/html/qemu-devel/2012-11/msg02697.html
- new qmp/hmp command for the state of each dimm (on/off)
Changes v2->v3
- qdev integration. Dimms are attached to a dimmbus. The dimmbus is a child
of i440fx device in the pc machine. Hot-add and remove are done with normal
device_add / device_del operations on the dimmbus. New commands "dimm_add" and
"dimm_del" are obsolete.
- Add _PS3 method to allow OSPM-induced hot operations.
- pci-window calculation in Seabios takes dimms into account(for both 32-bit and
64-bit windows)
- rename new qmp commands: query-memory-total and query-memory-hotplug
- balloon driver can see the hotplugged memory
Changes v1->v2
- memory map is automatically calculated for hotplug dimms. Dimms are added from
top-of-memory skipping the pci hole at [PCI_HOLE_START, 4G).
- Renamed from "-memslot" to "-dimm". Commands changed to "dimm_add", "dimm_del"
- Seabios ejection array reduced to a byte. Use extraction macros for dimm ssdt.
- additional SRAT paravirt info does not break previous SRAT fw_cfg layout.
- Documentation of new acpi_piix4 registers and paravirt data.
- add ACPI _OST support for _OST enabled guests. This allows qemu to receive
notification for success / failure of memory hot-add and hot-remove operations.
Guest needs to support _OST (https://lkml.org/lkml/2012/6/25/321)
- add monitor info command to report total guest memory (initial + hot-added)
Issues:
- hot-remove needs to only unmap the dimm device from guest's view. Freeing the
memory should happen when the last user of the device (e.g. virtio-blk) unrefs
the device. A testcase is needed for this.
- Live Migration: Ramblocks are migrated before qdev VMStates are migrated. So
the DimmDevice is handled diferrently than other devices. Should this be
reworked ?( DimmDevice structure currently does not define a VMStateDescription)
Live migration works as long as the dimm layout (command line args) are
identical at the source and destination qemu command line, and destination takes
into account hot-operations that have occured on source. (v3 patch 10/19
created the DimmDevice that corresponds to an unknown incoming ramblock, e.g.
for a dimm that was hot-added on source. but has been dropped for the moment).
- A main blocker issue is windows guest functionality. The patchset does not
work for windows currently. Testing on win2012 server RC or windows2008
consumer prerelease, when adding a DIMM, there is a BSOD with ACPI_BIOS_ERROR
message. After this, the VM keeps rebooting with ACPI_BIOS_ERROR. The windows
pnpmem driver obviosuly has a problem with the seabios dimm implementation
(or the seabios dimm implementation is not fully ACPI-compliant). If someone
can review the seabios patches or has any ideas to debug this, let me know.
- hot-operation notification lists need to be added to migration state.
series is based on:
- qemu master (commit a8a826a3) + patch:
https://lists.gnu.org/archive/html/qemu-devel/2012-11/msg02699.html
- seabios master (commit a810e4e7)
Can also be found at:
http://github.com/vliaskov/qemu-kvm/commits/memhp-v4
http://github.com/vliaskov/seabios/commits/memhp-v4
Vasilis Liaskovitis (21):
qapi: make visit_type_size fallback to type_int
Add SIZE type to qdev properties
qemu-option: export parse_option_number
Implement dimm device abstraction
vl: handle "-device dimm"
acpi_piix4 : Implement memory device hotplug registers
acpi_ich9 : Implement memory device hotplug registers
piix_pci and pc_piix: refactor
piix_pci: Add i440fx dram controller initialization
q35: Add i440fx dram controller initialization
pc: Add dimm paravirt SRAT info
Introduce paravirt interface QEMU_CFG_PCI_WINDOW
Implement "info memory-total" and "query-memory-total"
balloon: update with hotplugged memory
Implement dimm-info
dimm: add hot-remove capability
acpi_piix4: add hot-remove capability
acpi_ich9: add hot-remove capability
Implement qmp and hmp commands for notification lists
Add _OST dimm support
Implement _PS3 for dimm
docs/specs/acpi_hotplug.txt | 54 ++++++
docs/specs/fwcfg.txt | 28 +++
hmp-commands.hx | 6 +
hmp.c | 41 ++++
hmp.h | 3 +
hw/Makefile.objs | 2 +-
hw/acpi.h | 5 +
hw/acpi_ich9.c | 115 +++++++++++-
hw/acpi_ich9.h | 12 +-
hw/acpi_piix4.c | 126 ++++++++++++-
hw/dimm.c | 444 +++++++++++++++++++++++++++++++++++++++++++
hw/dimm.h | 102 ++++++++++
hw/fw_cfg.h | 1 +
hw/lpc_ich9.c | 2 +-
hw/pc.c | 28 +++-
hw/pc.h | 1 +
hw/pc_piix.c | 74 ++++++--
hw/pc_q35.c | 18 ++-
hw/piix_pci.c | 249 ++++++++-----------------
hw/q35.c | 27 +++
hw/q35.h | 5 +
hw/qdev-properties.c | 60 ++++++
hw/qdev-properties.h | 3 +
hw/virtio-balloon.c | 13 +-
monitor.c | 21 ++
qapi-schema.json | 63 ++++++
qapi/qapi-visit-core.c | 11 +-
qemu-option.c | 4 +-
qemu-option.h | 4 +
qmp-commands.hx | 57 ++++++
sysemu.h | 1 +
vl.c | 60 ++++++
32 files changed, 1432 insertions(+), 208 deletions(-)
create mode 100644 docs/specs/acpi_hotplug.txt
create mode 100644 docs/specs/fwcfg.txt
create mode 100644 hw/dimm.c
create mode 100644 hw/dimm.h
Vasilis Liaskovitis (9):
Add ACPI_EXTRACT_DEVICE* macros
Add SSDT memory device support
acpi-dsdt: Implement functions for memory hotplug
acpi: generate hotplug memory devices
q35: Add memory hotplug handler
pci: Use paravirt interface for pcimem_start and pcimem64_start
acpi: add _EJ0 operation and eject port for memory devices
Add _OST dimm method
Implement _PS3 method for memory device
Makefile | 2 +-
src/acpi-dsdt-mem-hotplug.dsl | 136 +++++++++++++++++++++++++++++++++++
src/acpi-dsdt.dsl | 5 +-
src/acpi.c | 158 +++++++++++++++++++++++++++++++++++++++--
src/paravirt.c | 6 ++
src/paravirt.h | 2 +
src/pciinit.c | 9 +++
src/q35-acpi-dsdt.dsl | 6 +-
src/ssdt-mem.dsl | 73 +++++++++++++++++++
tools/acpi_extract.py | 28 +++++++
10 files changed, 415 insertions(+), 10 deletions(-)
create mode 100644 src/acpi-dsdt-mem-hotplug.dsl
create mode 100644 src/ssdt-mem.dsl
--
1.7.9