[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH v3] spapr: Add a new level of NUMA for GPUs
From: |
David Gibson |
Subject: |
Re: [PATCH v3] spapr: Add a new level of NUMA for GPUs |
Date: |
Thu, 16 Jul 2020 15:04:59 +1000 |
On Mon, May 25, 2020 at 12:49:27PM -0500, Reza Arbab wrote:
> On Mon, May 25, 2020 at 03:05:50PM +1000, David Gibson wrote:
> > On Fri, May 22, 2020 at 02:53:33PM -0500, Reza Arbab wrote:
> > > --- a/hw/ppc/spapr_pci_nvlink2.c
> > > +++ b/hw/ppc/spapr_pci_nvlink2.c
> > > @@ -362,7 +362,7 @@ void spapr_phb_nvgpu_ram_populate_dt(SpaprPhbState
> > > *sphb, void *fdt)
> > > uint32_t associativity[] = {
> > > cpu_to_be32(0x4),
> > > SPAPR_GPU_NUMA_ID,
> > > - SPAPR_GPU_NUMA_ID,
> > > + cpu_to_be32(nvslot->numa_id),
> > > SPAPR_GPU_NUMA_ID,
> > > cpu_to_be32(nvslot->numa_id)
> >
> >
> > This doesn't look quite right. In the new case we'll get {
> > GPU_NUMA_ID, nvslot->numa_id, GPU_NUMA_ID, nvslot->numa_id }.
>
> The associativity reference points are 4 (and now 2), so this is what we
> want. I think what you've noticed is that reference points are 1-based
> ordinals:
>
> "...the “ibm,associativity-reference-points” property indicates
> boundaries between associativity domains presented by the
> “ibm,associativity” property containing “near” and “far” resources. The
> first such boundary in the list represents the 1 based ordinal in the
> associativity lists of the most significant boundary, with subsequent
> entries indicating progressively less significant boundaries."
Right.. AIUI, associativity-reference-points indicates which leves are
"important" from a NUMA distance point of view (though the spec is
very confusing). But, I'm pretty sure, that ignoring
reference-points, the individual ibm,associativity lists are supposed
to describe a correct hierarchy, even if some levels will get ignored
for distance purposes. So once you've split up into "numa_id" nodes
at the second level, you can't then go back to just 2 nodes (main
vs. gpu) at the third.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
signature.asc
Description: PGP signature
- Re: [PATCH v3] spapr: Add a new level of NUMA for GPUs,
David Gibson <=