Re: [Qemu-devel] [PATCH] vfio: Align iova also to IOMMU page size

[Pavel Fedin]

 Hello!

> >  So, i've got this problem on ARM64. On ARM64 we actually can never have 1K 
> > pages. This page
> > size was supported only by old 32-bit ARM CPUs, up to ARMv5 IIRC, then it 
> > was dropped. Linux
> > OS never even used it.
> >  But, since qemu can emulate those ancient CPUs, TARGET_PAGE_BITS is 
> > defined to 10 for ARM.
> > And, ARM64 and ARM32 is actually the same target for qemu, so this is why 
> > we still get it.
> >  Perhaps, TARGET_PAGE_BITS should be a variable for ARM, and we should set 
> > it according to
> > the actual used CPU. Then this IOMMU alignment problem would disappear 
> > automatically. What do
> > you think?
> >  Cc'ed Peter since he is the main ARM guy here.
> 
> Do we only see these alignments when we're emulating those old 1k page
> processors?

 No. We see them when we are emulating brand new ARM64. And we see them only 
because TARGET_PAGE_BITS is still hardcoded to 10.

> If not, should we really be telling a 4k page VM about 1k
> aligned memory?

 Well, first of all, we are not telling about aligned memory at all. I've 
researched for the origin of this address, it appears when we are mapping MSI-X 
BAR of the VFIO device.
 My device defines this BAR to be of 2M size. In this case qemu splits it up 
into three regions:
 1) Region below the MSI-X table (it's called "mmap", for me it's empty because 
table offset is 0)
 2) MSI-X table itself (20 vectors = 0x00000140 bytes for me).
 3) Region above the MSi-X table (it's called "mmap msix-hi"). Size for my case 
is 2M - REAL_HOST_PAGE_ALIGN(0x140) = 0x1FF000.
Regions (1) and (3) are passed through and directly mapped by VFIO, region (2) 
is emulated.
 Now, we have PBA. The device says that PBA is placed in memory specified by 
the same BAR as table, and its offset is 0x000f0000. PBA is also emulated by 
qemu, and as a result it overlaps "mmap msix-hi" region, which breaks up into 
two fragments as a consequence:
 3a) offset 0x00000 size 0x0EF000
 3b) offset 0xEF008 size 0x10FFF8
 PBA sits between these fragments, having only 8 bytes in size.
 Attempt to map (3b) causes the problem. vfio_mmap_region() is expected to 
align this up, and it does, but it does it to a minimum possible granularity 
for ARM platform, which, as qemu thinks, is always 1K.

>  If that's all legit, maybe we should be aligning down
> rather than up, we know the host memory is at least 4k pages, so
> anything in the gap between those alignments should be backed by memory,
> right?

 You know, i also have this question. Why are we aligning up? What if the 
device tries to access parts that we skipped by our alignment?

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia