[RFC] Implementing RLIMIT_AS

[Diego Nieto Cid]

Hello,

After playing a bit with the setrlimit calls in Linux to see how the
resource RLIMIT_AS actually works, it seems to be limiting the amount
of virtual memory available to the process (and not the available
virtual address range).

So, I'm attempting to implement this limit in gnumach at the points
memory is allocated. The patch below contains the first few steps
I took in this attempt:

  1. Add a `hard_limit` field, of type size_t, to the vm_map struct.
     I started with hard limit beacause I still need to research how
     soft limits work. So, for now, it's a plain rejection with ENOMEM.

  2. At vm_map_setup, initialize the `hard_limit` field with the
     appropriate value which should be RLIM_INFINITY. In this first
     attempt, I hardcoded 8GB for testing.

     I tried a lower value, like 2GB, but some process is mapping
     4GB at once during boot and it just hangs when the allocation
     fails.

  3. Finally, enforce the limit in `vm_allocate`, `vm_map` and
     `vm_allocate_contiguous` by checking that current map size
     (`size` field) plus the requested size (`size` param) is less
     than the current map's `hard_limit` field.

I thought of adding an RPC call that sets the `hard_limit` field
which, I guess, should be located among the other task related RPCs.

Also, I wanted to ask whether I covered all the allocation points
or there is somewhere else where limits shall be enforced. For instance,
something I still have to look at is the out-of-line data sent through
a mach_msg call.

Regards

----

Index: gnumach-1.8+git20240714/vm/vm_map.c
===================================================================
--- gnumach-1.8+git20240714.orig/vm/vm_map.c
+++ gnumach-1.8+git20240714/vm/vm_map.c
@@ -198,6 +198,9 @@ void vm_map_setup(
        map->first_free = vm_map_to_entry(map);
        map->hint = vm_map_to_entry(map);
        map->name = NULL;
+       /* TODO hardcoded limit for testing purposes, rather use RLIM_INFINITY 
*/
+       /* TODO add RPC to update this limit */
+       map->hard_limit = 8l * 1024l * 1024l * 1024l;
        vm_map_lock_init(map);
        simple_lock_init(&map->ref_lock);
        simple_lock_init(&map->hint_lock);
Index: gnumach-1.8+git20240714/vm/vm_map.h
===================================================================
--- gnumach-1.8+git20240714.orig/vm/vm_map.h
+++ gnumach-1.8+git20240714/vm/vm_map.h
@@ -198,6 +198,10 @@ struct vm_map {
        unsigned int            timestamp;      /* Version number */
 
        const char              *name;          /* Associated name */
+
+       /* TODO only hard limits are enforced */
+       /* TODO does getting rlim_t here make sense? */
+       vm_size_t               hard_limit;     /* hard limit as set by 
RLIMIT_AS */
 };
 
 #define vm_map_to_entry(map)   ((struct vm_map_entry *) &(map)->hdr.links)
Index: gnumach-1.8+git20240714/vm/vm_user.c
===================================================================
--- gnumach-1.8+git20240714.orig/vm/vm_user.c
+++ gnumach-1.8+git20240714/vm/vm_user.c
@@ -81,6 +81,12 @@ kern_return_t vm_allocate(
                *addr = trunc_page(*addr);
        size = round_page(size);
 
+       if (map->size + size > map->hard_limit)
+         {
+               printf("map size: %lu, requested size: %lu, hard limit: %lu\n", 
map->size, size, map->hard_limit);
+               return(KERN_NO_SPACE);
+         }
+
        result = vm_map_enter(
                        map,
                        addr,
@@ -348,6 +354,13 @@ kern_return_t vm_map(
        *address = trunc_page(*address);
        size = round_page(size);
 
+       if (target_map->size + size > target_map->hard_limit)
+         {
+               printf("map size: %lu, requested size: %lu, hard limit: %lu\n", 
target_map->size, size, target_map->hard_limit);
+               return(KERN_NO_SPACE);
+         }
+
+
        if (!IP_VALID(memory_object)) {
                object = VM_OBJECT_NULL;
                offset = 0;
@@ -631,6 +644,12 @@ kern_return_t vm_allocate_contiguous(
        if (size == 0)
                return KERN_INVALID_ARGUMENT;
 
+       if (map->size + size > map->hard_limit)
+         {
+               printf("map size: %lu, requested size: %lu, hard limit: %lu\n", 
map->size, size, map->hard_limit);
+               return(KERN_NO_SPACE);
+         }
+
        object = vm_object_allocate(size);
 
        if (object == NULL)