Re: VMM

[Farid Hajji]

> My proposal is to design the VMM in the same way we have designed the
> other components of the Hurd, that is, in a distributed fashion.  I
> have come up with a model where there is a central physical memory
> server and swap server (however, they can be stacked: it is only a
> question of a server providing an appropriate interface).  All
> processes compete for the memory on the system and enter in to
> different types of contracts with the physical memory server.  They
> can then use the memory anyway they like but when the contract runs
> our, they must return the memory if asked.  If a server ignores these
> request, the memory server can revoke all of its mappings and
> effectively kill the client.

1. The physical memory server will need an algorithm to decide:
   * which pages to revoke once memory becomes scarce
   * what resources to allocate to clients, without risking
     starving those clients (e.g. wired memory, etc...)

So, basically, the physical memory server will need to implement
parts of the VMM's strategies for dealing with clients.

2. Competing VM managers (possibly with non-compatible
   strategies) that access a physical memory pool (or server)
   can have conflicting goals. This behavior has not been carefully
   studied yet (AFAIK. If you know of any paper, please let me know).

It may be wise to keep strategies out of a physical memory server.
Consider UVM: The whole system uses a physical memory management
module called pmap, which implements a policy-free interface.
[Actually, pmap is very easy to implement in L4]. The decisions
which pages to hand out, which ones to page out/in and which ones
to evict, are not pmap's but UVMs.

Actually, separating the physical memory server from the (multiple!)
VM servers would lead to more inefficiencies w.r.t. zeroing memory:
In an integrated pmap+VM system, the VM system knows when zeroed-
pages are needed and when uninitialized pages would be enough.
In the distributed case, the 'pmap' would always need to zero
pages before handing them over to the competing VM servers (for
obvious security reasons). That's a lot more cycles to the memory
bus. Because the phys-server would probably be accessed very often
by VM servers that do aggressive memory allocation/deallocation
[of scratch buffers e.g.]; that would lead to a _lot_ of unnecessary
zeroing. Hmmm....

> As tasks must manage their own memory, they must make page eviction
> decisions, etc.  But this is generally a good thing: we would prefer
> that the client make the decisions as it knows the most about its
> memory usage patterns and what it will and will not be using in the
> near term.  It may even be able to decide to just drop temporary
> scratch buffers.  This type of decision would be using a monolithic
> system.  Instead, the monolithic VMM would be forced to swap the data
> to disk.  The same holds true at allocation time: the task knows what
> the memory will be used for, as such, it can ask for certain page
> sizes or not worry about zero filling memory.  The end result is a lot
> less guessing and better utilization of memory.  Well, at least in
> theory.

Sure, clients know best what to do which their memory.
BUT, they'll still compete with other clients, who have
conflicting goals.

The "magic" of VM really lives in the VM module/task/phys-server/...
which resolves the conflicts by determining which pages are more
recently used, etc. No single client can do this, because it simply
doesn't see the "big picture". It all boils do to this: You partition
the memory space into sub-spaces that would be individually managed
and locally optimzied, but you won't get global optimization here.

Basically, if you allocate a "fixed" amount of physical memory to
clients, this would almost always be sub-optimial:
  * Clients would get too much (wasting) or not enough memory
    and would need to handshake with the phys-server anyway.
  * Clients generally can't estimate the amount of memory
    at startup time, so they'll have to adapt their physical
    memory resources to their needs (contacting phys-server
    to release or to require pages), and this all the time
    they're running. [Lot's of IPC ahead]
  * In case of memory shortage, page eviction becomse necessary(+)
  * I'm not even starting to think about demand-paging,
    mmap(2) etc... in this context!

+) This is an interesting point: Consider an algorithm by which
the mem-server IPCs the clients, requiring them to clean up some
pages and return them to the mem-server. Some clients may be able
to do so, others won't and some may be buggy. So finally, it will
still be mem-server's call to evict more pages, should the pages
that are returned on client's free will not be enough... etc.

> Another mitigating factor in the design is avoiding having servers
> provide resources to clients.  That is, servers provide services and
> nothing more; clients should provide any required resources.  In the
> case of a reading from a file, for instance, this would entail
> providing memory to the server which could fill it with appropriate
> data.  Memory "containers" allow memory to be passed around logically
> and also permits memory to be locked by untrusted servers without the
> client losing control over it.  Other mechanisms allow the tasks to
> make logical copies of memory.  This would be useful in the case of a
> file system's page cache.

Well, if you make the physical memory server the primary VM and add
2nd-level (sub-)VM servers down the chain, it would probably work.
Yet it remains to be proven that the added complexity pays off in
terms of efficiency. I highly doubt that it will. Just one example:

Monolithic kernels, e.g. BSD, used to have multiple memory allocators
for various parts of the system. Those allocators used memory from
different pools. The result was always an under-utilization (and
therefore wasting) of memory. This is not the case anymore with
newer kernels [don't confuse this kernel-only use with user-land
VM usage; but the point still holds]

Think also about the benefits that resulted from the merge of VM
and buffer cache. Sure, the result is an ugly mess of dependencies,
yet it is blindingly fast. Moreover, it's practical too, not only
for mmap(2) & friends. Here again, unifying the memory requirements
into a single VM management space (like UVM, VM+Buffercache, etc...)
is a Good Thing(tm), because of the "Big Picture" mentioned earlier.
The system simply adapts better to global requirements and adjusts
itself more gracefully.

I know that this is counter-intuitive. I also prefer clean interfaces,
separated responsibilities etc..., but in the case of VM, this
doesn't seem to improve the situation. Actually, the opposite
is the case. Very strange, but true. That was the most amazing
lesson I've learned on VM principles!

-Farid.

-- 
Farid Hajji -- Unix Systems and Network Admin | Phone: +49-2131-67-555
Broicherdorfstr. 83, D-41564 Kaarst, Germany  | address@hidden
- - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - -
Due to budget cuts, light at end of tunnel will be out. --Unknown.