Re: More than 255 vcpus Windows VM setup without viommu ?

[David Woodhouse]

On Mon, 2024-07-08 at 10:13 +0100, David Woodhouse wrote:
> On Wed, 2024-07-03 at 16:01 +0000, Sandesh Patel wrote:
> > > 
> > > Interesting. What exactly has Windows *done* in those MSI entries?
> > > That might give a clue about how to support it.

I repeated my experiment with interrupt-only remapping (no DMA
remapping). On two hosts:



vendor_id       : AuthenticAMD
cpu family      : 25
model           : 17
model name      : AMD EPYC 9R14 96-Core Processor

vendor_id       : GenuineIntel
cpu family      : 6
model           : 143
model name      : Intel(R) Xeon(R) Platinum 8488C

I used identical command lines on both, and on each host I got the same
result with all of '-cpu host', '-cpu EPYC' and -cpu Skylake-Server'.
It's the *host* that makes the difference, not the CPUID presented to
the guest.

On the Intel host it boots:

$ ./qemu-system-x86_64 -accel kvm,kernel-irqchip=split -cdrom 
~/Win10_22H2_EnglishInternational_x64v1.iso  -m 16G -M q35 -smp 
2,cores=12,threads=2,maxcpus=288 -accel kvm,kernel-irqchip=split -device 
intel-iommu,intremap=on,dma-translation=off -cpu Skylake-Server --trace 
vtd_ir_\* --trace apic\* --trace kvm_irqchip\*  
qemu-system-x86_64: -accel kvm,kernel-irqchip=split: warning: Number of 
hotpluggable cpus requested (288) exceeds the recommended cpus supported by KVM 
(192)
kvm_irqchip_add_msi_route dev N/A vector 0 virq 0
kvm_irqchip_add_msi_route dev N/A vector 0 virq 1
kvm_irqchip_add_msi_route dev N/A vector 0 virq 2
kvm_irqchip_add_msi_route dev N/A vector 0 virq 3
kvm_irqchip_add_msi_route dev N/A vector 0 virq 4
kvm_irqchip_add_msi_route dev N/A vector 0 virq 5
kvm_irqchip_add_msi_route dev N/A vector 0 virq 6
kvm_irqchip_add_msi_route dev N/A vector 0 virq 7
kvm_irqchip_add_msi_route dev N/A vector 0 virq 8
kvm_irqchip_add_msi_route dev N/A vector 0 virq 9
kvm_irqchip_add_msi_route dev N/A vector 0 virq 10
kvm_irqchip_add_msi_route dev N/A vector 0 virq 11
kvm_irqchip_add_msi_route dev N/A vector 0 virq 12
kvm_irqchip_add_msi_route dev N/A vector 0 virq 13
kvm_irqchip_add_msi_route dev N/A vector 0 virq 14
kvm_irqchip_add_msi_route dev N/A vector 0 virq 15
kvm_irqchip_add_msi_route dev N/A vector 0 virq 16
kvm_irqchip_add_msi_route dev N/A vector 0 virq 17
kvm_irqchip_add_msi_route dev N/A vector 0 virq 18
kvm_irqchip_add_msi_route dev N/A vector 0 virq 19
kvm_irqchip_add_msi_route dev N/A vector 0 virq 20
kvm_irqchip_add_msi_route dev N/A vector 0 virq 21
kvm_irqchip_add_msi_route dev N/A vector 0 virq 22
kvm_irqchip_add_msi_route dev N/A vector 0 virq 23
kvm_irqchip_commit_routes 
qemu-system-x86_64: warning: host doesn't support requested feature: 
CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: 
CPUID.07H:EBX.rtm [bit 11]
qemu-system-x86_64: warning: host doesn't support requested feature: 
CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: 
CPUID.07H:EBX.rtm [bit 11]
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
vtd_ir_enable enable 1
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
vtd_ir_remap_msi_req addr 0xfee00010 data 0xd1 sid 0xffff do_fault 0
vtd_ir_irte_get index 0 low 0x0 high 0x100d10005
vtd_ir_remap index 0 trigger 0 vector 209 deliver 0 dest 0x1 mode 1
vtd_ir_remap_type IOAPIC
vtd_ir_remap_msi (addr 0xfee00010, data 0xd1) -> (addr 0xfee01004, data 0x40d1)
kvm_irqchip_update_msi_route Updating MSI route virq=2
kvm_irqchip_commit_routes 
vtd_ir_remap_msi_req addr 0xfee00010 data 0xd1 sid 0xffff do_fault 0
vtd_ir_irte_get index 0 low 0x0 high 0x100d10005
vtd_ir_remap index 0 trigger 0 vector 209 deliver 0 dest 0x1 mode 1
vtd_ir_remap_type IOAPIC
vtd_ir_remap_msi (addr 0xfee00010, data 0xd1) -> (addr 0xfee01004, data 0x40d1)
kvm_irqchip_update_msi_route Updating MSI route virq=2
kvm_irqchip_commit_routes 
...


On the AMD host it stops at the boot splash screen:

$ ./qemu-system-x86_64 -accel kvm,kernel-irqchip=split -cdrom 
~/Win10_22H2_EnglishInternational_x64v1.iso  -m 16G -M q35 -smp 
2,cores=12,threads=2,maxcpus=288 -accel kvm,kernel-irqchip=split -device 
intel-iommu,intremap=on,dma-translation=off -cpu  Skylake-Server --trace 
vtd_ir_\* --trace apic\* --trace kvm_irqchip\* 
qemu-system-x86_64: -accel kvm,kernel-irqchip=split: warning: Number of 
hotpluggable cpus requested (288) exceeds the recommended cpus supported by KVM 
(192)
kvm_irqchip_add_msi_route dev N/A vector 0 virq 0
kvm_irqchip_add_msi_route dev N/A vector 0 virq 1
kvm_irqchip_add_msi_route dev N/A vector 0 virq 2
kvm_irqchip_add_msi_route dev N/A vector 0 virq 3
kvm_irqchip_add_msi_route dev N/A vector 0 virq 4
kvm_irqchip_add_msi_route dev N/A vector 0 virq 5
kvm_irqchip_add_msi_route dev N/A vector 0 virq 6
kvm_irqchip_add_msi_route dev N/A vector 0 virq 7
kvm_irqchip_add_msi_route dev N/A vector 0 virq 8
kvm_irqchip_add_msi_route dev N/A vector 0 virq 9
kvm_irqchip_add_msi_route dev N/A vector 0 virq 10
kvm_irqchip_add_msi_route dev N/A vector 0 virq 11
kvm_irqchip_add_msi_route dev N/A vector 0 virq 12
kvm_irqchip_add_msi_route dev N/A vector 0 virq 13
kvm_irqchip_add_msi_route dev N/A vector 0 virq 14
kvm_irqchip_add_msi_route dev N/A vector 0 virq 15
kvm_irqchip_add_msi_route dev N/A vector 0 virq 16
kvm_irqchip_add_msi_route dev N/A vector 0 virq 17
kvm_irqchip_add_msi_route dev N/A vector 0 virq 18
kvm_irqchip_add_msi_route dev N/A vector 0 virq 19
kvm_irqchip_add_msi_route dev N/A vector 0 virq 20
kvm_irqchip_add_msi_route dev N/A vector 0 virq 21
kvm_irqchip_add_msi_route dev N/A vector 0 virq 22
kvm_irqchip_add_msi_route dev N/A vector 0 virq 23
kvm_irqchip_commit_routes 
qemu-system-x86_64: warning: host doesn't support requested feature: 
CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: 
CPUID.07H:EBX.rtm [bit 11]
qemu-system-x86_64: warning: This family of AMD CPU doesn't support 
hyperthreading(2). Please configure -smp options properly or try enabling 
topoext feature.
qemu-system-x86_64: warning: host doesn't support requested feature: 
CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: 
CPUID.07H:EBX.rtm [bit 11]
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
vtd_ir_enable enable 1
kvm_irqchip_commit_routes 
kvm_irqchip_commit_routes 
vtd_ir_remap_msi_req addr 0xfee00004 data 0x0 sid 0xffff do_fault 0
vtd_ir_remap_msi (addr 0xfee00004, data 0x0) -> (addr 0xfee00004, data 0x0)
kvm_irqchip_update_msi_route Updating MSI route virq=2
kvm_irqchip_commit_routes 
^Cqemu: terminating on signal 2


It looks like Windows is putting something bogus into the IOAPIC RTE for irq2?
In the successful boot that was 0xfee00010 / 0xd1; vector 209 on CPU0.

smime.p7s
Description: S/MIME cryptographic signature