async thread creation

[Johan Rydberg]

Marcus and I had a discussion earlier tonight on #hurd.  This mail
contains a few followup questions to Marcus.

First to summarize what we discussed:

There's a dead-lock situation when creating worker threads for
physmem.  The situation arises when the task server has to allocate
memory to create a new thread, and that thread is to become a worker
thread for physmem.  The task server will then contact physmem to
request more memory, but physmem has to worker threads to serve the
request with.  Physmem will then try to create a new worker thread by
contacting the task server.  Bang! Boom! Dead-lock!

Marcus and I discussed several solutions to this problem.  

One solution would be to have a fixed number of pre-allocated worker
threads for physmem.  If the RPC server runs out of worker threads it
will block and wait for a worker thread to become available.  This
idea was dismissed since it would put a too heavy limit on the server.

Another idea was to have a dedicated worker thread for communication
between the task server and physmem.  This idea was rejected because
of some argument I can't remember right now. 

The third solution involved having a separate single-threaded RPC
management (server) thread, just serving requests from the task
server.  But it would be troublesome to share data structures between
the dedicated task RPC manager and the multi-threaded RPC manager that
servers all other requests.

It would also be possible to have a separate thread that performs
worker thread allocation.  When ever the server runs out of worker
threads, it contacts the worker creator thread and lets it know that
the server is all out of threads.  The creator thread will in turn
contact the task server (through a regular RPC) and create a new
thread and insert it into the worker thread tool.  If the task server
tries to contact physmem while creating the new thread, the RPC
manager thread would block until a worker thread becomes available.
Sooner or later a worker thread will become free, serving task's
request.  Progress will be made, till the point where the new worker
thread is fully created and inserted into the thread pool.

What is critical in this design, and different from the other
solutions described above, is that physmem is allowed to keep
processing RPC requests (making it possible for the task server to
communicate with physmem while creating a new worker thread,) modulo
creating new worker threads.

Also, the (worker) thread doing the initial request for more workers
may continue with its task.  When done, it will put itself back on the
pool.  So there's a _big_ chance there's a free working thread if the
task server decides to contact physmem for more memory (provided that
there's isn't too much RPC pressure on physmem.)

This is what I calls the async-worker-thread-creation policy.  I hope
I did not miss anything.



Was your (Marcus) idea that this async-worker-thread-creation policy
should only be implemented in physmem, or as a general part of the RPC
facility?

If the latter is true, the question is how often this scenario would
arise.  I would imagine that after a while the number of worker
threads would stabilize.  What worries me is bursts of RPC requests.

To stress a server you could simply spawn a large number of threads
and then do parallel RPC's to the server.  But if you do that, I'm
pretty sure you would put larger constrains on the system, and the
starvation of the target server would be a small problem in
comparison.

I would prefer if we did not have any special cases for task/physmem,
especially when it comes to fundamental things such as RPC handling
and management.  I understand how hard it will be to accomplish this,
though.

~j