multiple capabilities in a single RPC

[Neal H. Walfield]

We need an interface to allow multiple capability objects to be
accessed by a single RPC stub.  This is required, for instance, by the
container interface where physical memory is logically copied between
two containers.  Consider:

  error_t container_copy (container_t source, uintptr_t s_start,
                          container_t dest, uintptr_t d_start,
                          size_t size)

container_copy logically copies the SIZE bytes of physical memory in
container SOURCE starting at byte S_START to container DEST starting
at byte D_START.


Right now, an RPC is executed if MR1 contains a capability handle
which can be used to successfully looked up a capability object in the
context of the bucket and sender.  We need to extend
libhurd-cap-server to provide a mechanism to access capability objects
associated with capability handles in registers other than MR1.  There
are a two ways that come to mind: message tagging (either inline or
externally based on the message ID) or expanding the
libhurd-cap-server API.  I don't want to even think about the hair
involved with the former.

My first reaction is to propose:

  error_t hurd_cap_obj_lookup (hurd_cap_bucket_t bucket,
                               hurd_task_id_t task_id,
                               hurd_cap_handle_t handle,
                               hurd_cap_obj_t *r_obj)


As you rightfully pointed out in a private discussion, this is
deceivingly simple: we need to consider if the calling thread should
be registered as a pending RPC in which case the worker thread is
registered twice in the bucket and client objects and potentially the
class and object arguments.  This can have some interesting effects
when one of these in inhibited (and the pending RPCs on it are
canceled).  For instance, it introduces a race: if the thread is
cancelled and immediately terminates after the first pthread_cancel
and the pthread id is reused before the second instance is come
across, we will inadvertently cancel a new unrelated thread.

This interface also means we gratuitously lookup the client structure
which is a shame but this is O(1) so it is tolerable.

It does mean, however, that we check the inhibited state of (at least)
the bucket and client twice which means taking a few locks.  This may
be a performance penalty we have to live with.  But maybe we can limit
the scope of this function and only allow it to be called from threads
executing an RPC on another capability in bucket BUCKET and from
TASK_ID thereby eliding the need to check the inhibition state and
removing the need to add the thread to the pending RPC lists.  (If
both capability objects are of the same class, the problem remains but
we might be able to come up with another interesting work around.)

We also have to consider what happens at the end of the RPC: the
function which called hurd_cap_obj_lookup must also call a function to
remove itself from pending rpc lists (at least the capability object
it looked up) and drop some references, etc.

Maybe a better interface would then be:

  error_t hurd_cap_obj_start_rpc (hurd_cap_bucket_t bucket,
                                  hurd_task_id_t task_id,
                                  hurd_cap_handle_t handle,
                                  hurd_cap_obj_t *r_obj)

  void hurd_cap_obj_end_rpc (hurd_cap_obj_t *obj)

with the constraint that the bucket BUCKET and the client associated
with task ID TASK_ID have already been verified to not be inhibited
and the current thread has been added to their pending rpc lists. We'd
need some magic for the class and the capability object.  At least
with respect to the capability object, we could potentially put the
burden on the caller to check that the capability is not a duplicate
of something it already looked up.

Ideas?  Reactions?

Thanks,
Neal