Re: Memory again

[Stefan Monnier]

>> - Many memory problems are misdiagnosed as fragmentation problems, where
>> the real cause is either inefficient memory use, or memory leaks.
>> - Real fragmentation problems exist but are fairly rare.
> It would be nice to have a built-in (optionally selected at configure time)
> method to provide a 'shot' of the heap to see how it's (mis)used. It's also
> interesting whether it's possible to write a set of gdb macros for doing
> something similar.

`garbage-collect' is supposed to give that info.  At least it does give
info about the data that's kept under alloc.c's control because
of fragmentation (these are counted as "free cells").

>> - In some "fragmentation" cases, the Emacs process holds on to memory
>> without any good reason, i.e. it's been "free"d but the malloc library
>> does not want or can't return it to the OS because it did not use mmap
>> to allocate it.  This can be fixed, but this memory would really be
>> unused; it can still appear in RSS but only because the memory
>> pressure hasn't yet bumped it out of RAM.  IOW these cases may show
>> high memory use in terms of VSZ or RSS but fixing them is low priority
>> because their only direct cost is use of of swap space.

> IIUC, this is not true, at least for Linux (see how zap_pte_range() updates
> MM_ANONPAGES RSS counter; it's done when munmap() happens). Unmapped
> (but still resident in RAM) pages aren't accounted as RSS of any process;
> they're accounted separately and amount of space occupied by such pages is
> 'Active(anon)' in /proc/meminfo.

Re-read what I wrote: I'm talking about the case where alloc.c free'd the
data, but malloc did not munmap it.

>> - Fixing the remaining real fragmentation problems probably requires
>> a different allocator that can move objects to compact the memory
>> space.  Maybe such an allocator can be retrofitted into Emacs
>> (e.g. a mostly-copying allocator), but noone has tried it yet
>> (probably because the outcome is hard to predict, the problem it
>> attempts to fix only occurs rather rarely, and it can be difficult to
>> ensure that it doesn't affect negatively the more common cases).

> It's not so hard to predict the memory-saving benefits of copying or
> compacting collector - ideally such a collector should free everything

It's easy to predict what it does in terms of "what is the benefit when
I have a lot of fragmentation".  It's much more difficult to predict
what it does in terms of "how is it going to affect Emacs's behavior for
those 99% cases where it currently works just fine".

> hold of a free lists is ~3-5% of total heap size; it's reasonable to expect
> that copying/compacting collector may also decrease an overall heap
> fragmentation, which may give a few more percents. Anyway, ~5% isn't
> convincing enough to start a hard work on a new GC; instead, I believe

I see we agree.

> that some minor optimization of current algorithm and Lisp data
> representation (vector allocation, compact or 'immediate' strings and
> something like cdr-coding or unrolled lists) may give comparable, or
> even better, effect in the sense of memory consumption.

I agree that we're probably going to see better overall results by
improving general memory use than by trying to attack
fragmentation problems.


        Stefan