Re: [Qemu-devel] How do we do user input bitmap properties?

[Andrew Jones]

On Thu, Apr 18, 2019 at 03:03:02PM +0100, Dave Martin wrote:
> On Thu, Apr 18, 2019 at 12:28:47PM +0100, Andrew Jones wrote:
> > On Thu, Apr 18, 2019 at 11:52:04AM +0100, Dave Martin wrote:
> > > On Thu, Apr 18, 2019 at 10:46:34AM +0100, Andrew Jones wrote:
> > > > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote:
> > > > > Hi all,
> > > > > 
> > > > > First some background:
> > > > > 
> > > > > For the userspace side of AArch64 guest SVE support we need to
> > > > > expose KVM's allowed vector lengths bitmap to the user and allow
> > > > > the user to choose a subset of that bitmap. Since bitmaps are a
> > > > > bit awkward to work with then we'll likely want to expose it as
> > > > > an array of vector lengths instead. Also, assuming we want to
> > > > > expose the lengths as number-of-quadwords (quadword == 128 bits
> > > > > for AArch64 and vector lengths must be multiples of quadwords)
> > > > > rather than number-of-bits, then an example array (which will
> > > > > always be a sequence) might be
> > > > > 
> > > > >  [ 8, 16, 32 ]
> > > > > 
> > > > > The user may choose a subsequence, but only through truncation,
> > > > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is.
> > > > > 
> > > > > Furthermore, different hosts may support different sequences
> > > > > which have the same maximum. For example, if the above sequence
> > > > > is for Host_A, then Host_B could be
> > > > > 
> > > > >  [ 8, 16, 24, 32 ]
> > > > 
> > > > It doesn't really matter for this discussion, but I just realized
> > > > that I picked bogus numbers for these examples. 32 would be too
> > > > big. The largest supported is 16. I probably should have just used
> > > > the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to
> > > > vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512.
> > > 
> > > I'd argue differently from the ABI perspective.
> > > 
> > > If the host supports vq = [ 1,2,3,4 ], then it is entirely valid to ask
> > > for [ 1,2,4 ].  KVM will fail the KVM_REG_ARM64_SVE_VLS write with
> > > EINVAL, but I don't distinguish between a set that is not satisfiable on
> > > this hardware and a set that is not valid at all.  The kernel basically
> > > doesn't check for the latter: an architecturally invalid set will always
> > > not be satisfiable on the hardware.
> > 
> > I agree that 1,2,4 should be a valid set, but how do we avoid the user
> > attempting to select it on a platform that supports 1,2,3,4 now? Until
> > the hardware and KVM will allow it, then I don't think we should try
> > to "support" it in any way. Maybe we should name the property such that
> > it's clear we need to use truncation when constructing a subset now.
> > This would allow us to add another, more generally named, property later
> > when we can select nearly arbitrary subsets.
> > 
> > > 
> > > Ideally, userspace should not preempt this decision, in case the
> > > architecture becomes more flexible someday in what it can virtualise.
> > > 
> > > > > The host must support all lengths in the sequence, which means
> > > > > that while Host_A supports 32, since it doesn't support 24 and
> > > > > we can only truncate sequences, we must use either [ 8 ] or
> > > > > [ 8, 16 ] for a compatible sequence if we intend to migrate
> > > > > between the hosts.
> > > > > 
> > > > > Now to the $SUBJECT question:
> > > > > 
> > > > > My feeling is that we should require the sequence to be
> > > > > provided on the command line as a cpu property. Something
> > > > > like
> > > > > 
> > > > >   -cpu host,sve-vl-list=8:16
> > > 
> > > I can't decide whether it's more or less user-friendly to denote VL in
> > > terms of bytes in this kind of context.
> > 
> > I'm not sure either. Switching to my second take at quadwords this should
> > have been '1:2'. In bytes [ 1, 2 ] would be '16:32'.
> > 
> > > 
> > > Usually when I say "VL" in the kernel ABI I try to mean this.
> > > (KVM_REG_ARM64_SVE_VLS is a special case: here the lengths are fancily
> > > encoded rather than being passed as integers, so the precise unit used
> > > is a more abstract concept here ... or anyway, that's my excuse.
> > > _SVE_VQS felt odd here.)
> > 
> > My thinking is that using the number of quadwords keeps the numbers
> > smaller. But, if people are mostly going to want to think about vector
> > lengths in terms of bits, like the specification does, then maybe we
> > should just let the numbers be bigger: [ 128, 256, 512 ]
> > 
> > > 
> > > For kvmtool I went with quadwords on the command line, but only as a
> > > quick hack to simplify the parser slightly.
> > > 
> > > OTOH, specifying 1,2,4 on the command line is clearly less annoying and
> > > error-prone than 16,32,64.
> > > 
> > > Naming the option with "vq" instead of "vl" is another option, though
> > > "vq" is Linux terminology not endorsed by the architecture.
> > 
> > I think VL is OK as long we document the unit.
> 
> As you wish (and that's basically where I ended up for now with kvmtool).
> 
> > > 
> > > For kvmtool I've considered a range / filtering syntax, but I didn't
> > > implement it yet.
> > 
> > I think we want each VL to be explicitly listed on the command line
> > for the compatibility issue described above. On one host the range
> > [1,4] can be different than the range [1,4] on anther host, depending
> > on whether '3' is in there. And syntax like 1-4:-3 would be overly
> > complicated, IMO. So I'm inclined to just require all of them to be
> > listed, but of course order wouldn't matter.
> 
> Probably reasonable.  For now (and quite likely forever) the length of
> the list is at most 16 elements.  To avoid overengineering I stuck with
> a flat comma-separated list for now.
> 
> 
> It's worth nothing that there are two problems to be solved here: one is
> to specify an exact set unambiguously, which is important for migration
> scenarios.
> 
> The other is to be able to clamp the vector length for user convenience,
> but without particularly considering migration.  There's no direct way
> for the user to know the set of vector lengths supported by KVM today,
> so cooking up the right magic to go on the command line is non-trivial:
> you have to create a dummy vcpu and read KVM_REG_ARM64_SVE_VLS to find
> the host's supported set.  Or you just have to know.

Right, we'll have to query first to know what's available. libvirt will
do that and tools built on libvirt should help provide the user with
sane defaults or a relatively painless selection process. The QEMU command
line is primarily for developers, so they'll likely just know what they
want.

> 
> A separate max-vl option (or whatever) might be a useful alternative.

Yeah, we could offer this to be nicer to QEMU command line users that
don't intend to migrate, or don't care if a migration can fail. This
would be analogous to '-cpu max' which enables all available cpu
features. Bad for migration, but a nice shorthand if you want your
guest to have everything.

Thanks,
drew