Re: [PATCH v5 13/13] docs/specs: add riscv-iommu

[Daniel Henrique Barboza]

On 7/19/24 6:34 AM, Alistair Francis wrote:
> On Tue, Jul 9, 2024 at 3:37 AM Daniel Henrique Barboza
> <dbarboza@ventanamicro.com> wrote:
> > Add a simple guideline to use the existing RISC-V IOMMU support we just
> > added.
> >
> > This doc will be updated once we add the riscv-iommu-sys device.
> >
> > Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>
> > ---
> >   docs/specs/index.rst       |  1 +
> >   docs/specs/riscv-iommu.rst | 55 ++++++++++++++++++++++++++++++++++++++
> >   docs/system/riscv/virt.rst | 13 +++++++++
> >   3 files changed, 69 insertions(+)
> >   create mode 100644 docs/specs/riscv-iommu.rst
> >
> > diff --git a/docs/specs/index.rst b/docs/specs/index.rst
> > index 1484e3e760..c68cd9ae6c 100644
> > --- a/docs/specs/index.rst
> > +++ b/docs/specs/index.rst
> > @@ -33,3 +33,4 @@ guest hardware that is specific to QEMU.
> >      virt-ctlr
> >      vmcoreinfo
> >      vmgenid
> > +   riscv-iommu
> > diff --git a/docs/specs/riscv-iommu.rst b/docs/specs/riscv-iommu.rst
> > new file mode 100644
> > index 0000000000..fa38ff7667
> > --- /dev/null
> > +++ b/docs/specs/riscv-iommu.rst
> > @@ -0,0 +1,55 @@
> > +.. _riscv-iommu:
> > +
> > +RISC-V IOMMU support for RISC-V machines
> > +========================================
> > +
> > +QEMU implements a RISC-V IOMMU emulation based on the RISC-V IOMMU spec
> > +version 1.0 [1].
> > +
> > +The emulation includes a PCI reference device, riscv-iommu-pci, that QEMU
> > +RISC-V boards can use.  The 'virt' RISC-V machine is compatible with this
> > +device.
> > +
> > +A platform device that implements the RISC-V IOMMU will be added in the
> > +future.
> > +
> > +
> > +riscv-iommu-pci reference device
> > +--------------------------------
> > +
> > +This device implements the RISC-V IOMMU emulation as recommended by the section
> > +"Integrating an IOMMU as a PCIe device" of [1]: a PCI device with base class 
> > 08h,
> > +sub-class 06h and programming interface 00h.
> > +
> > +As a reference device it doesn't implement anything outside of the 
> > specification,
> > +so it uses a generic default PCI ID given by QEMU: 1b36:0014.
> > +
> > +To include the device in the 'virt' machine:
> > +
> > +.. code-block:: bash
> > +
> > +  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci (...)
>
> We should add a sentence saying what this does. As in what should a
> user expect after they have done this

Here's what was added at this point:

 .. code-block:: bash
 
-  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci (...)
+  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci,[optional_pci_opts] 
(...)
+
+This will add a RISC-V IOMMU PCI device in the board following any additional
+PCI parameters (like PCI bus address).  The behavior of the RISC-V IOMMU is
+defined by the spec but its operation is OS dependent.  As of this writing the
+existing Linux kernel support `linux-v8`_, not yet merged, will configure the 
IOMMU
+to create IOMMU groups with any eligible cards available in the system,
+regardless of factors such as the order in which the devices are added in the
+command line.
+
+This means that these command lines are equivalent as far as the current
+IOMMU kernel driver behaves:
+
+.. code-block:: bash
+
+  $ qemu-system-riscv64 \
+        -M virt,aia=aplic-imsic,aia-guests=5 \
+        -device riscv-iommu-pci,addr=1.0,vendor-id=0x1efd,device-id=0xedf1 \
+        -device e1000e,netdev=net1 -netdev user,id=net1,net=192.168.0.0/24 \
+        -device e1000e,netdev=net2 -netdev user,id=net2,net=192.168.200.0/24 \
+        (...)
+
+  $ qemu-system-riscv64 \
+        -M virt,aia=aplic-imsic,aia-guests=5 \
+        -device e1000e,netdev=net1 -netdev user,id=net1,net=192.168.0.0/24 \
+        -device e1000e,netdev=net2 -netdev user,id=net2,net=192.168.200.0/24 \
+        -device riscv-iommu-pci,addr=1.0,vendor-id=0x1efd,device-id=0xedf1 \
+        (...)
+
+Both will create iommu groups for the two e1000e cards.


What I tried to say here is that the operation of the IOMMU device is OS 
dependent,
and the current Linux driver doesn't care to things such as the order in which 
the
devices were added. Is this what you had in mind?


I also removed the platform device reference at the start of the doc. I think 
it was
doing more harm than good. We'll need to change this doc when that device is 
added,
might as well leave it as is for now.


Thanks,

Daniel


> Alistair
>
> > +
> > +As of this writing the existing Linux kernel support [2], not yet merged, is 
> > being
> > +created as a Rivos device, i.e. it uses Rivos vendor ID.  To use the 
> > riscv-iommu-pci
> > +device with the existing kernel support we need to emulate a Rivos PCI IOMMU by
> > +setting 'vendor-id' and 'device-id':
> > +
> > +.. code-block:: bash
> > +
> > +  $ qemu-system-riscv64 -M virt        \
> > +     -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1 (...)
> > +
> > +Several options are available to control the capabilities of the device, 
> > namely:
> > +
> > +- "bus"
> > +- "ioatc-limit"
> > +- "intremap"
> > +- "ats"
> > +- "off" (Out-of-reset translation mode: 'on' for DMA disabled, 'off' for 
> > 'BARE' (passthrough))
> > +- "s-stage"
> > +- "g-stage"
> > +
> > +
> > +[1] 
> > https://github.com/riscv-non-isa/riscv-iommu/releases/download/v1.0/riscv-iommu.pdf
> > +[2] 
> > https://lore.kernel.org/linux-riscv/cover.1718388908.git.tjeznach@rivosinc.com/
> > diff --git a/docs/system/riscv/virt.rst b/docs/system/riscv/virt.rst
> > index 9a06f95a34..8e9a2e4dda 100644
> > --- a/docs/system/riscv/virt.rst
> > +++ b/docs/system/riscv/virt.rst
> > @@ -84,6 +84,19 @@ none``, as in
> >
> >   Firmware images used for pflash must be exactly 32 MiB in size.
> >
> > +riscv-iommu support
> > +-------------------
> > +
> > +The board has support for the riscv-iommu-pci device by using the following
> > +command line:
> > +
> > +.. code-block:: bash
> > +
> > +  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci (...)
> > +
> > +Refer to :ref:`riscv-iommu` for more information on how the RISC-V IOMMU 
> > support
> > +works.
> > +
> >   Machine-specific options
> >   ------------------------
> >
> > --
> > 2.45.2