Re: [Qemu-devel] [RFC] Device sandboxing

[Corey Bryant]

On 12/07/2011 01:48 PM, Anthony Liguori wrote:
> On 12/07/2011 12:25 PM, Corey Bryant wrote:
> > A group of us are starting to work on sandboxing QEMU device emulation
> > code. We're just getting started investigating various approaches, and
> > want to engage the community to gather input.
> >
> > Following are the design points that we are currently considering:
>
> To be perfectly honest, I think prototyping and measuring performance is
> going to be the only way to figure out the right approach here. Here are
> some thoughts on the various approaches.
>
> > * Decompose QEMU into multiple processes:
> >
> > * This could be done such that QEMU devices execute in separate
> > processes based on device type, e.g. all block devices in one
> > process and all network devices in a second process. Another
> > alternative is executing a separate process per device.
>
> I don't think that a HIRD of QEMU-replacing daemons is the best approach
> to this problem. While I appreciate the academic attraction to such a
> proposal, I think practical experience tells us that this isn't the
> easiest type of system to get right.

Thanks for the input.

The idea would be to fork() the processes internally, if that is the 
concern.  They wouldn't have to be started separately by the user.

> > * Decomposition would not only afford a level of security inherent
> > in process separation, it would also allow development of stricter
> > sVirt/SELinux policy for the decomposed QEMU processes (e.g. a
> > block device specific policy). This would enable a true sandbox
> > with layers of defense.
> >
> > * Decompose the device emulation process further into an untrusted and
> > trusted thread:
>
> I think this general approach is the most rationale place to start.

Agreed.

> > * The untrusted thread would be restricted by seccomp mode 1 and
> > would contain the device emulation code.
>
> I think the best strategy would allow for a device to run either in the
> untrusted thread or the trusted thread. This makes performance testing a
> bit easier and it also makes development a bit more natural.

When you refer to the device running in the trusted thread, are you 
talking about the case where you run QEMU without sandboxing support?  I 
think we would ideally like to add this new support such that if it is 
not enabled, QEMU will still run as a single process and decomposition 
wouldn't occur.

> > * The trusted helper thread would run beside the untrusted thread,
> > enabling the untrusted thread to make syscalls beyond read(),
> > write(), exit(), and sigreturn().
>
> I assume you mean process, not thread BTW?

I do mean thread.  When making calls on behalf of the seccomp'd thread, 
I think there will be syscalls that must be called from the same address 
space.  That's where the the trusted helper thread would come into play.

> > * IPC communication mechanisms:
> >
> > * An IPC mechanism will be required to enable communication between
> > untrusted and trusted threads.
> >
> > * An IPC mechanism will also be required to enable communication
> > between the main QEMU process and device processes.
>
> IPC is easy. We have tons of infrastructure in QEMU for IPC (virtio,
> QMP, etc.). Please don't reinvent the wheel here.

Ok

> > * The communication mechanisms must provide secure communication,
> > be low overhead (easy to generate, parse, and validate), and must
> > play well with sVirt/LSMs.
>
> I don't see how sVirt/LSM fits into this but all of these requirements
> are also true for the other big untrusted thread that we interact with
> (the guest itself).
>
> My view is that we should view the untrusted thread as an extension of
> the guest and that the interfaces between the trusted thread and the
> untrusted thread views it simply as another machine type that presents a
> different (simpler) hardware abstraction.

Yes this makes sense.  I think our biggest concern with IPC is that we 
don't introduce a TOCTTOU opportunity for a device to change call 
parameters after they've been checked and before the calls is made on 
behalf of the sandboxed thread.  Shared memory that is writable by both 
untrusted/trusted thread could introduce this.

> > * Some thoughts for IPC mechanisms are Unix sockets, pipes, virtio,
> > Google Native Client's IMC, and shared memory.
>
> The actual mechanism doesn't really matter I think, but see above comments.
>
> > * If seccomp mode 2 support becomes available, decomposition of device
> > emulation into untrusted/trusted threads may not be necessary. This
> > could result in improved performance (no IPC overhead between trusted
> > and untrusted thread) and reduced complexity (no need for trusted
> > helper thread).
>
> If mode 2 is the Right Answer, then we shouldn't wait for it to become
> available. We should make it available by pushing it into the kernel.
>
> If we all agree that if mode 2 existed, it's what we would use, then
> that we have the answer to this discussion and we know what we need to
> go off and do.

That would seem like the logical approach.  I think there may be new 
mode 2 patches coming soon so we can see how they go over.

> > * Execution of QEMU with the sandboxed device support should be an
> > optional run-time specification.
>
> Ack with a small exception. If we can demonstrate that sandboxing has an
> acceptable performance overhead, then we should do it unconditionally to
> reduce our overall test matrix. It's unclear that that's obtainable though.

Good point.

> > * We will be focusing on legacy devices first, both for performance and
> > risk reasons.
> >
> > Once we settle on a direction, we will develop a proof of concept to
> > share with the community.
>
> Proof of concepts are the only way to settle on direction. Code speaks
> louder than anything else.

Definitely.

> > We appreciate your input.
> >
> > Regards,
> >
> > Ashley Lai
> > Corey Bryant
> > Eduardo Otubo
> > Michael Halcrow
> > Paul Moore
> > Richa Marwaha
>
> In the future, I would suggest beginning these type of discussions on
> the list to start with. Otherwise valuable in information (including
> discussion and debate on directions) are not available to the greater
> community at large.
>
> Not a big deal in this case, but I want to be on the record here about
> this. I would have greatly preferred this whole effort start out on
> qemu-devel from day one.

Understood.

> Regards,
>
> Anthony Liguori
>
> >

--
Regards,
Corey