Re: [Qemu-devel] [PATCH v2 05/24] numa: move source of default CPUs to NUMA node mapping into boards

[Igor Mammedov]

On Thu, 4 May 2017 17:32:13 +1000
David Gibson <address@hidden> wrote:

> On Wed, May 03, 2017 at 11:42:40AM -0300, Eduardo Habkost wrote:
> > On Wed, May 03, 2017 at 02:56:59PM +0200, Igor Mammedov wrote:  
> > > Originally CPU threads were by default assigned in
> > > round-robin fashion. However it was causing issues in
> > > guest since CPU threads from the same socket/core could
> > > be placed on different NUMA nodes.
> > > Commit fb43b73b (pc: fix default VCPU to NUMA node mapping)
> > > fixed it by grouping threads within a socket on the same node
> > > introducing cpu_index_to_socket_id() callback and commit
> > > 20bb648d (spapr: Fix default NUMA node allocation for threads)
> > > reused callback to fix similar issues for SPAPR machine
> > > even though socket doesn't make much sense there.
> > > 
> > > As result QEMU ended up having 3 default distribution rules
> > > used by 3 targets /virt-arm, spapr, pc/.
> > > 
> > > In effort of moving NUMA mapping for CPUs into possible_cpus,
> > > generalize default mapping in numa.c by making boards decide
> > > on default mapping and let them explicitly tell generic
> > > numa code to which node a CPU thread belongs to by replacing
> > > cpu_index_to_socket_id() with @cpu_index_to_instance_props()
> > > which provides default node_id assigned by board to specified
> > > cpu_index.
> > > 
> > > Signed-off-by: Igor Mammedov <address@hidden>  
> > 
> > Reviewed-by: Eduardo Habkost <address@hidden>
> > 
> > Just two extra comments below:
> > 
> > [...]  
> > > +static CpuInstanceProperties
> > > +virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
> > > +{
> > > +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> > > +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
> > > +
> > > +    assert(cpu_index < possible_cpus->len);
> > > +    return possible_cpus->cpus[cpu_index].props;;
> > > +}
> > > +  
> > [...]  
> > > +static CpuInstanceProperties
> > > +pc_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
> > >  {
> > > +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> > > +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
> > > +
> > > +    assert(cpu_index < possible_cpus->len);
> > > +    return possible_cpus->cpus[cpu_index].props;;
> > >  }  
> > 
> > The fact that these two implementations look exactly the same
> > made me wonder:
> > 
> > 1) Why this isn't the default implementation;
> > 2) Why exactly spapr needs a different implementation.
> > 
> > Then I noticed that there's nothing in the common machine code
> > that specifies that possible_cpus->cpus[] is indexed by
> > cpu_index. This means it is indeed safer to require each machine
> > to provide its own cpu_index_to_props implementation than having
> > a default implementation that can unexpectedly break (e.g. if
> > granularity at possible_cpus is not at VCPU/thread level).
> > 
> > I would still like to have an abstraction that wouldn't require
> > writing machine-specific code (e.g. cpu_index ranges to
> > possible_cpus like David suggested), but that's for a follow-up
> > series.  
> 
> Yeah, that similarity bothered me to, but like you I realised the
> problem is that spapr simply doesn't have the same granularity of
> information as x86 and ARM - there's only one entry per core for PAPR
> instead of one per thread.
> 
> So, we do need a machine specific mapping of cpu_index to location
> properties, which is what the callback is for.
> 
> It does occur to me that another way of accomplishing that would be
> for possible_cpu_arch_ids() to create a cpu_index->props mapping as a
> simple array ofpointers, in addition to the list of possiblee props
> structures.
> 
> Not sure if that would end up looking better or not.
I'd like to remove foo_cpu_index_to_props() callbacks in future,
to do so we would need to obsolete and remove cpu_index based
'-numa node,cpus='. Lets give a year for new interface to settle
in and then remove old interface.

so I'd rather do not extend possible_cpus with more complex structure
and related mgmt routines.



> 
> > [...]  
> > > +static CpuInstanceProperties
> > > +spapr_cpu_index_to_props(MachineState *machine, unsigned cpu_index)
> > >  {
> > > +    CPUArchId *core_slot;
> > > +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> > > +
> > > +    /* make sure possible_cpu are intialized */
> > > +    mc->possible_cpu_arch_ids(machine);
> > > +    core_slot = spapr_find_cpu_slot(machine, cpu_index, NULL);
> > > +    assert(core_slot);
> > > +    return core_slot->props;
> > >  }  
> > 
> > If you need to submit v3, maybe a comment here explaining why
> > spapr needs a different cpu_index_to_props implementation would
> > be helpful. I took a while to figure it out.
> >   
>