Re: Proposal: block-based vector allocator

[Stefan Monnier]

>> Someone could try to change the allocator so those blocks are exactly
>> 1-page in size and get allocated via mmap:
> I did this even before I started with block-based vector allocator.

How much did it help?

>> - using mmap will ensure that we factor out malloc from the equation.
>> - using page-size (presumably 4KB) blocks rather than the current 16KB
> Current sizes are 1K except string data, which is 8K.

For cons cells and floats, we call malloc for 16KB blocks at a time
(which we then cut into 1024 blocks).  This is because we need the 1024
blocks to be aligned on a 1024 boundary (so when we use malloc instead
of posix_memalign, we waste upto 1023 (well 1016, really) bytes of those
16KB).

> My favorite byte-force-recompile benchmark usually consumes ~89M. With my
> vector allocator it shrinks to 78M.

And you said that using the non-conservative stack scanning resulted in
similar results, so all that extra memory is occupied by mem_nodes (or
fragments that used to hole mem_nodes ;-).

> But for (definitely pathological) compile buffer case, results are
> very unexpected (I tried with 100000-lines buffer).  Fresh instance
> is 18M.  With compile buffer, it grows to 142M and shrinks to 141M
> after killing *compilation* and garbage-collect.  When mmap() is enabled,
> it grows to 138M and shrinks back to 21M (twenty one) after kill & GC.

Hmm.. cool.  So some of the responsability for this test to behave so
poorly might be in malloc rather than in alloc.c.  Good news.
Oh, yes, now I remember:

#ifdef DOUG_LEA_MALLOC
      /* Prevent mmap'ing the chunk.  Lisp data may not be mmap'ed
         because mapped region contents are not preserved in
         a dumped Emacs.  */
      mallopt (M_MMAP_MAX, 0);
#endif

So maybe malloc behaves poorly simply because we don't let it
behave better.


        Stefan