Re: thread ids, task ids and subsystems

[Espen Skoglund]

[Niels Möller]
> Espen Skoglund <address@hidden> writes:
>>> On Thu, Apr 10, 2003 at 04:39:20PM +0200, Niels Möller wrote:
>>>> [...]
>>>> I'd expect L4 to schedule B on cpu 1, because (1) the thread that
>>>> was running there was just put to sleep, so the cpu is free to
>>>> take, and (2) the RPC arguments are located in *registers* inside
>>>> cpu 1.
>> 
>> ...and the whole cache working set of B is (perhaps) on cpu 0.
>> Writing some register contents to a cache line and "transferring"
>> that cache line to the destination cpu is far outweighed by the
>> cost of transferring the thread's working set (stack, data, etc.)
>> *from* the other cpu.  Then there's the additional cost of
>> synchronizing scheduling, dealing with migration, etc.

> Does this mean that if the server keeps one thread per cpu, it's
> best for clients to send the rpc requests to a thread running on a
> *different* cpu? You may save some working set migration, if you're
> lucky enough to target the right cpu, but on the other you need
> inter-cpu communication and both cpu:s are forced to do some
> scheduling.

Eh... why would it make sense to target a server thread on another
cpu.  If there is a server thread on the current cpu it definitely
pays of to send the rpc to this one (no working set migration, not
inter-cpu communication, direct switching to the server thread).

> If I understood Marcus right, the idea was to target the rpc to a
> server thread on the *same* cpu as the client thread. Was that
> wrong? 

No.  I suppose I must have read your text out of context (I haven't
been following this thread too closely :-).

> Another subtlety with the one-thread per cpu model is that ports and
> reply ports must be treated differently. Consider a Hurd rpc that is
> implemented as two L4 rpc calls, one for the request and another for
> the reply.

> 1. Thread A wants to make a request to task B, so it sends an rpc to
>    one of B's threads running on a particular cpu (confused again, so
>    I'm not sure now if it's the same cpu as the one A is running on,
>    or a different).

> 2. B starts to serve the request, and while doing that it might even
>    be rescheduled to a different cpu.

> 3. B sends the reply. It must send the reply to the thread A, no other
>    thread in the same task is good enough.

> So service ports will sometimes or always be served by multiple
> threads, one per cpu, while a reply ports must always be a single
> thread.

(Considering that I possibly have completely misunderstood what you're
talking about here, you might want take my answer with a grain of
salt.)

There is of course no must that a service port have one thread on each
cpu.  Doing so, however, can greatly increase performance since you
avoid cross cpu communication (i.e., cross cpu RPCs).  Also, your
second bullet above mention that B might be migrated to a different
cpu.  Such a migration will not be initiaded by the L4 kernel,
however, and I see no reason why the Hurd runtime would choose to do
such a migration either (unless perhaps thread B does not occupy all
cpus and the system wants to do some load balancing).

As for the reply port, it really depens on how you implement it.  If
you do a synchronous reply (i.e., A is blocked waitng for the reply),
the "reply port" might best be implemented as a direct "A does a
receive from B", and hence the "reply port" is the thread A itself.
If you do an asynchronous scheme, you'll probably have some sort of
proxy to receive the reply.  This proxy could itself be implemented as
a set of threads located on different cpus.

        eSk