Re: [Qemu-devel] [PATCH 09/16] Enable message delivery via IRQs

[Blue Swirl]

On Sat, Jun 12, 2010 at 7:33 PM, Blue Swirl <address@hidden> wrote:
> On Sat, Jun 12, 2010 at 3:58 PM, Paul Brook <address@hidden> wrote:
>>> I think message passing interrupts
>>> are only used in bus based systems, like PCI or UPA/JBUS etc. I don't
>>> know how LAPIC/IOAPIC bus works, it could be similar.
>>
>> PCI Message Signalled Interrupts use a regular data write, and we model it
>> exactly that way. Under normal circumstances you program the device to write
>> to memory mapped APIC control registers, but there's no real reason why you
>> couldn't write to RAM.
>>
>> LAPIC/IOAPIC have their own bus. On some hardware this is multiplexed over 
>> the
>> main system bus, but logically it is a separate interconnect.
>>
>> The fact that these a bus based systems suggests against using qemu_irq, 
>> which
>> is an inherently point-point system.
>>
>>> > How is a
>>> > receiver meant to know for format of the message being delivered, and who
>>> > it's intended for?
>>>
>>> This depends on the architecture. On Sparc64 the message is specified
>>> by the guest or OF.
>>>...
>>> In real HW (MSI or Sparc64 mondo interrupts) a bus delivers the
>>> message. But our buses only handle address and data lines.
>>
>> IIUC you're trying to use qemu_irq as a generic interconnect between devices.
>> I think that's going to cause more problems than it solves.  If a bus has
>> additional message passing capabilities then this can be part of the bus
>> interface.
>>
>> If two devices need a private communication channel then we probably want 
>> some
>> implementation agnostic way of defining this link.  Connecting LAPIC to 
>> IOAPIC
>> is a potential example of this. However this needs to be done in a type-safe
>> manner.  DMA channels may be another potential use of this linking. We'd want
>> to avoid connecting a DMA channel to an APIC.
>>
>> [*] A simple unidirectional dma request line is suitable for qmu_irq. A DMA
>> system that transfers data outside of memory read/write transactions is not.
>> e.g. ISA effectively defines a regular memory bus plus 8 bidirectional data
>> streams (aka DMA channels). These are multiplexed over the same physical 
>> pins,
>> but logically distinct. The DMA channels could either be bundled up into the
>> ISA bus interface, or modelled as links between devices and the DMA
>> controller.
>
> Very very interesting. There's some out of band data related to DMA
> (bus errors, IOMMU faults, ISA DREQ) which hasn't been covered in
> previous Generic DMA proposals. Maybe all we need is a generic side
> band link, which would be usable for point to point (qemu_irq,
> coalescing, messages) and bus channels?

I think we could solve all problems (well, maybe not world peace, yet)
by switching to message based system for all of DMA and IRQs.

Each device would have a message input port and way to output messages.

Examples:

Zero copy memory access from device D1 to D2 to host memory (D3) with
access broken to page length units and errors occurring on the last
byte:
D1 send_msg(ID, MSG_MEM_WRITE, DMA address, length) -> D2
D2 send_msg(ID, MSG_MEM_WRITE, DMA address2, length) -> D3
D3 send_replymsg(ID, MSG_MEM_PTR, host address, 4096) -> D2
D2 send_replymsg(ID, MSG_MEM_PTR, host address, 4096) -> D1
D3 send_replymsg(ID, MSG_MEM_PTR, host address, length - 4096 - 1) -> D2
D2 send_replymsg(ID, MSG_MEM_PTR, host address, length - 4096 - 1) -> D1
D3 send_replymsg(ID, MSG_MEM_ERROR, DMA address2 + length - 1, 1, status) -> D2
D2 send_replymsg(ID, MSG_MEM_ERROR, DMA address + length - 1, 1, status) -> D1

IRQ delivery chain D1->D2->D3 with coalescing, messages, delivery
reporting and EOI:
D1 send_msg(ID, MSG_IRQ_RAISE, payload) -> D2
D2 send_msg(ID, MSG_IRQ_RAISE, payload) -> D3
D3 send_replymsg(ID, MSG_IRQ_STATUS, MI_COALESCED) -> D2
D2 send_replymsg(ID, MSG_IRQ_STATUS, MI_COALESCED) -> D1
D3 send_replymsg(ID, MSG_IRQ_STATUS, MI_DELIVERED) -> D2
D2 send_replymsg(ID, MSG_IRQ_STATUS, MI_DELIVERED) -> D1
D3 send_replymsg(ID, MSG_IRQ_STATUS, MI_EOI) -> D2
D2 send_replymsg(ID, MSG_IRQ_STATUS, MI_EOI) -> D1

What else do we want? :-)