[Qemu-devel] [RFC PATCH v3 0/3] Add support for NUMA on ARM64

[Shannon Zhao]

Add support for NUMA on ARM64. Tested successfully running a guest
Linux kernel with the following patch applied:

- arm64:numa: adding numa support for arm64 platforms.
http://article.gmane.org/gmane.linux.ports.arm.kernel/382179

Changes v2 ... v3:
* update to use NUMA node property arm,associativity.

Changes v1 ... v2:
Take into account Peter's comments:
* rename virt_memory_init to arm_generate_memory_dtb
* move arm_generate_memory_dtb to boot.c and make it a common func
* use a struct numa_map to generate numa dtb

Example qemu command line:
qemu-system-aarch64 \
    -enable-kvm -smp 4\
    -kernel Image \
    -m 512 -machine virt,kernel_irqchip=on \
    -initrd guestfs.cpio.gz \
    -cpu host -nographic \
    -numa node,mem=256M,cpus=0-1,nodeid=0 \
    -numa node,mem=256M,cpus=2-3,nodeid=1 \
    -append "console=ttyAMA0 root=/dev/ram"

Todo:
1)The NUMA nodes information in DT is not finalized yet, so this
patch might need to be further modified to follow any changes in it.

2)Consider IO-NUMA as well

Please refer to the following url for NUMA DT node details:

- Documentation: arm64/arm: dt bindings for numa.
http://article.gmane.org/gmane.linux.ports.arm.kernel/382180

Example: 2 Node system each having 2 CPUs and a Memory

        arm,associativity-reference-points = <0x0 0x1>;

        address@hidden {
                reg = <0x0 0x40000000 0x0 0xfffffff>;
                device_type = "memory";
                arm,associativity = <0x0 0x0 0xffff>;
        };
        address@hidden {
                reg = <0x0 0x50000000 0x0 0xfffffff>;
                device_type = "memory";
                arm,associativity = <0x1 0x0 0xffff>;
        };

        cpus {
                #size-cells = <0x0>;
                #address-cells = <0x1>;

                address@hidden {
                        reg = <0x0>;
                        enable-method = "psci";
                        compatible = "arm,arm-v8";
                        device_type = "cpu";
                        arm,associativity = <0x0 0x0 0x0>;
                };

                address@hidden {
                        reg = <0x1>;
                        enable-method = "psci";
                        compatible = "arm,arm-v8";
                        device_type = "cpu";
                        arm,associativity = <0x0 0x0 0x1>;
                };

                address@hidden {
                        reg = <0x2>;
                        enable-method = "psci";
                        compatible = "arm,arm-v8";
                        device_type = "cpu";
                        arm,associativity = <0x1 0x0 0x2>;
                };
                address@hidden {
                        reg = <0x3>;
                        enable-method = "psci";
                        compatible = "arm,arm-v8";
                        device_type = "cpu";
                        arm,associativity = <0x1 0x0 0x3>;
                };
        }

- arm,associativity

The mapping is done using arm,associativity device property.
this property needs to be present in every device node which needs to to be
mapped to numa nodes.

arm,associativity property is set of 32-bit integers. representing the
board id, socket id and core id.

ex:
        /* board 0, socket 0, core 0 */
        arm,associativity = <0 0 0x000>;

        /* board 1, socket 0, core 8 */
        arm,associativity = <1 0 0x08>;
        
- arm,associativity-reference-points

This property is a set of 32-bit integers, each representing an index into
the arm,associativity nodes. The first integer is the most significant
NUMA boundary and the following are progressively less significant boundaries.
There can be more than one level of NUMA.

Ex:
        arm,associativity-reference-points = <0 1>;
        The board Id(index 0) used first to calculate the associativity (node
        distance), then follows the  socket id(index 1).

        arm,associativity-reference-points = <1 0>;
        The socket Id(index 1) used first to calculate the associativity,
        then follows the board id(index 0).

        arm,associativity-reference-points = <0>;
        Only the board Id(index 0) used to calculate the associativity.

        arm,associativity-reference-points = <1>;
        Only socket Id(index 1) used to calculate the associativity.

Shannon Zhao (3):
  hw/arm/virt: Use memory_region_allocate_system_memory to allocate
    memory
  hw/arm/virt: Don't add memory node in creat_fdt
  hw/arm/boot: Add arm_generate_memory_dtb to generate memory dtb
    according to NUMA topology

 hw/arm/boot.c |   80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 hw/arm/virt.c |    7 +---
 2 files changed, 79 insertions(+), 8 deletions(-)