Re: Potential use case for opaque space bank: domain factored network stack

[Jonathan S. Shapiro]

On Sat, 2007-01-06 at 16:07 +0100, Marcus Brinkmann wrote:
> Hi,
> 
> we lost l4-hurd.  Feel free to reply there, I am leaving your mail
> intact so that nothing is lost.
> 
> At Sat, 06 Jan 2007 07:56:53 -0500,
> "Jonathan S. Shapiro" <address@hidden> wrote:
> > 
> > On Sat, 2007-01-06 at 02:07 +0100, Marcus Brinkmann wrote:
> > > Note that the network driver wants to do DMA
> > > and thus needs to be able to lock down the memory anyway, at least for
> > > a short period of time.  So, in a sense the TCP stack already
> > > dominates the user, it is part of the TCB.
> > 
> > In the design presented in "Reloaded", this is not the case. A separate
> > TCP/IP stack is instantiated per client, and the TCP/IP stack does NOT
> > dominate the client. The ethernet driver is shared by all TCP/IP stacks,
> > and (as with any driver capable of DMA) it is part of the system's
> > universal TCB. That is: *everything* relies on the ethernet driver,
> > whether they know it or not (because the ethernet driver is in a
> > position to overwrite the kernel).
> 
> There is something that is not clear to me.  Who (needs to) decide
> what is the code in the network stack implementation?  Is that
> provided by a user in the system as part of his shell, or by the
> system?  A closely related question is if the protocol between the
> stack and the driver is cooperative?

In the "Reloaded" design, the TCP/IP stack is assumed to be *completely*
untrusted. It is application code. The assumption of the ethernet driver
is that the TCP/IP stack is hostile. The assumption of the TCP/IP stack
is that the ethernet driver is reliable.

> > >>From a storage management perspective, the complication in "Reloaded"
> > comes from the intersection of two requirements:
> > 
> >   1. The ethernet driver needs to write into storage that is provided
> >      by the client.
> > 
> >   2. The client can revoke that storage.
> > 
> > The presence of DMA actually is not a big deal. The pages in question
> > are logical pages rather than physical pages. The ethernet can deal with
> > things in one of two ways:
> > 
> >   1. Extra copy. DMA to storage owned by ethernet and then copy to
> >      user storage. This is the simplest, but it is has performance
> >      problems.
> > 
> >      The only tricky bit here is that the ethernet must recover
> >      from a memory access violation if the user revokes the storage.
> >      This is a nuisance, but it isn't really that hard.
> > 
> >   2. Direct DMA. Ethernet performs privileged operations that cause
> >      the relevant user page to be placed in a DMA-accessible region
> >      of physical memory (moved by kernel if required). Prior to setting
> >      up DMA, Ethernet "pins" the page for DMA.
> > 
> >      A page pinned for DMA can be revoked, but the physical frame that
> >      contains that page at the time of revocation will not be released
> >      until it is unpinned for DMA.
> > 
> >      This design works great for *outbound* packets, but it is
> >      problematic for *inbound* packets, because packets are likely
> >      to end up in an input buffer (page) owned by the wrong party.
> >      The solution to this lies in three parts:
> > 
> >        1. Recognize that an ethernet packet is a public thing,
> >           so this temporary misplacement is not a security violation.
> > 
> >        2. Provide a "page exchange" operation so that this frame can
> >           be exchanged in place with the correct recipient.
> > 
> >      The remaining concern is that the initial, wrong, recipient might
> >      corrupt the packet (by writing the buffer) and thereby do denial
> >      of service. This is handled by ensuring that receive buffers are
> >      read-only to their TCP/IP stack. The space is provided by the stack
> >      (therefore revocable by the stack) but the address space is
> >      constructed by the ethernet (which therefore controls permissions)
> 
> This is not clear from the paper, which says that all regions are
> mapped RW into the stack and the driver.  Is this part of the pinning
> for DMA?

Yes. The version in the paper uses strategy (1). At the time, our
analysis suggested that copying large packets on the Pentium was likely
to be just as fast as page flipping. So the incoming strategy for the
version in the paper is to strip enough of the packet header to
determine the correct destination stack, and then copy the packet into
destination memory.

> > In practice, the correct strategy to use depends on the packet size, so
> > both would be used in a production implementation.
> > 
> > I want to draw attention to the last paragraph under (2). It raises a
> > case where ownership of resource (defined as: being the party who
> > allocates and pays) *MUST NOT* imply control over access to that
> > resource. The ability to separate these two things appears to be
> > intrinsic to avoiding denial of resource in communications involving
> > asymmetric trust relationships and large buffers.
> > 
> > > The core point here is that process A does not give any authority to
> > > process B that it didn't already have or could have (in principle).
> > 
> > I have lost track of who is A and who is B at this point, but I am not
> > convinced that this captures what is going on in this case adequately.
> 
> I may be abusing MarkM's terminology, so I will probably have to
> restate that.  My point is rather subtle and I am not sure I can
> explain it well.  The details also depend on the answer to the above
> question, so I will wait for that first, rather than exploring
> hypotheticals.
-- 
Jonathan S. Shapiro, Ph.D.
Managing Director
The EROS Group, LLC
+1 443 927 1719 x5100