Re: [RFC PATCH 16/25] hw/pxb/cxl: Add "windows" for host bridges

[Philippe Mathieu-Daudé]

On 11/13/20 1:49 AM, Ben Widawsky wrote:
> On 20-11-10 21:47:15, Ben Widawsky wrote:
>> In a bare metal CXL capable system, system firmware will program
>> physical address ranges on the host. This is done by programming
>> internal registers that aren't typically known to OS. These address
>> ranges might be contiguous or interleaved across host bridges.
>>
>> For a QEMU guest a new construct is introduced allowing passing a memory
>> backend to the host bridge for this same purpose. Each memory backend
>> needs to be passed to the host bridge as well as any device that will be
>> emulating that memory (not implemented here).
>>
>> I'm hopeful the interleaving work in the link can be re-purposed here
>> (see Link).
>>
>> An example to create a host bridges with a 512M window at 0x4c0000000
>>  -object memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M
>>  -device 
>> pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,len-memory-base=1,memory-base\[0\]=0x4c0000000,memory\[0\]=cxl-mem1
>>
>> Link: https://lists.nongnu.org/archive/html/qemu-devel/2020-08/msg03680.html
>> Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> 
> Hi Phil, wanted to call you out specifically on this one.
> 
> The newly released CXL 2.0 specification (which from a topology perspective 
> can
> be thought of as very PCIe-like) allows for interleaving of memory access.
> 
> Below is an example of two host bridges, each with two root ports, and 5 
> devices
> (two of switch are behind a switch).
> 
> RP: Root Port
> USP: Upstream Port
> DSP: Downstream Port
> Type 3 device: Memory Device with persistent or volatile memory.
> 
> +-------------------------+      +-------------------------+
> |                         |      |                         |
> |   CXL 2.0 Host Bridge   |      |   CXL 2.0 Host Bridge   |
> |                         |      |                         |
> |  +------+     +------+  |      |  +------+     +------+  |
> |  |  RP  |     |  RP  |  |      |  |  RP  |     |  RP  |  |
> +--+------+-----+------+--+      +--+------+-----+------+--+
>       |            |                   |               \--
>       |            |                   |        +-------+-\--+------+
>    +------+    +-------+            +-------+   |       |USP |      |
>    |Type 3|    |Type 3 |            |Type 3 |   |       +----+      |
>    |Device|    |Device |            |Device |   |                   |
>    +------+    +-------+            +-------+   | +----+     +----+ |
>                                                 | |DSP |     |DSP | |
>                                                 +-+----+-----+----+-+
>                                                     |          |
>                                                 +------+    +-------+
>                                                 |Type 3|    |Type 3 |
>                                                 |Device|    |Device |
>                                                 +------+    +-------+
> 
> Considering this picture... interleaving of memory access can happen in all 3
> layers in the topology.
> 
> - Memory access can be interleaved across host bridges (this is accomplished
>   based on the physical address chosen for the devices, those address ranges 
> are
>   platform specific and not part of the 2.0 spec, for now).
> 
> - Memory access can be interleaved across root ports in a host bridge.
> 
> - Finally, memory access can be interleaved across downstream ports.
> 
> I'd like to start the discussion about how this might overlap with the patch
> series you've last been working on to interleave memory. Do you have any
> thoughts or ideas on how I should go about doing this?

Main case:

 +-------------------------+
 |                         |
 |   CXL 2.0 Host Bridge   |
 |                         |
 |  +------+     +------+  |
 |  |  RP  |     |  RP  |  |
 +--+------+-----+------+--+
       |            |
       |            |
    +------+    +-------+
    |Type 3|    |Type 3 |
    |Device|    |Device |
    +------+    +-------+

// cxl device state
s = qdev_create(TYPE_CXL20_HB_DEV)

cxl_memsize = 2 * memsize(Type3Dev);

// container for cxl
memory_region_init(&s->container, OBJECT(s),
                   "container", cxl_memsize);

// create 2 slots, interleaved each 2k
s->interleaver = qdev_create(INTERLEAVER_DEV,
                             slotsize=2k,
                             max_slots=2)
qdev_prop_set_uint64(s->interleaver, "size",
                     cxl_memsize);

// connect each device to the interleaver
object_property_set_link(OBJECT(interleaver),
                         "mr0", OBJECT(RP0))
object_property_set_link(OBJECT(interleaver),
                         "mr1", OBJECT(RP1))
sysbus_realize_and_unref(SYS_BUS_DEVICE(interleaver))

// we can probably avoid this container
memory_region_add_subregion(&s->container, 0,
                            sysbus_mmio_get_region(interleaver, 0));


For the 2nd case, USP can be created the same way than case 1
(as a 2nd interleaver) then the main CXL is created with the
minor difference of mr1 now being the USP:

object_property_set_link(OBJECT(interleaver),
                         "mr1", OBJECT(USP))
sysbus_realize_and_unref(SYS_BUS_DEVICE(interleaver))

Regards,

Phil.