Re: Emacs bzr memory footprint

[Nix]

On 21 Oct 2011, Stefan Monnier said:

>> STIME   RSS    VSZ
>> Oct07 876524 1170572
>> Oct07 227016 499588
>
> I've just installed memory-usage.el as a package in the `elpa' branch
> (you'll soon be able to install it using package.el and in the mean time
> you can download it from 
> http://bzr.savannah.gnu.org/lh/emacs/elpa/download/head:/memoryusage.el-20111021130523-o1v7kfuzrat3pcd9-2/memory-usage.el).
>
> It basically looks at the garbage-collect output and the various
> buffer's size to provide you with a human-readable description of the
> memory in use, from Elisp's point of view.  Don't expect too much of it,
> but it can be helpful.
>
> Could you try it out?

I am actually generating a gigantic amount of debugging output at GC
time using a patch from Dmitry Antipov right now: in a few days once
it's bloated up we'll see what we can see.

I'll try memory-usage.el out too once the bloating really kicks in:
I was looking for it earlier but then remembered that it's XEmacs that
has 'memory-usage-stats' as an interactive subr :)

> If this output does not explain the process's size you're seeing, then
> we have a leak in the C code somewhere.  If it does, then we have either
> a leak in Elisp, or at least an excessive memory use by some package,
> and hopefully we can at least figure out which category of object
> is involved.

... or the same thing XEmacs apparently had, which was excessive memory
consumption by one or more other still-unidentified C libraries.

>> Unfortunately that means, sigh, using our own malloc() again, which is
>> probably more painful than useful.
>
> Emacs already uses its own malloc for most objects: it uses malloc
> directly to allocate vectors (and this is actually something that we
> might want to change, because it (indirectly) comes with an O(log n)
> cost), but for strings, conses, markers, overlays, and floats it uses
> malloc only to get a chunk of memory which it then manages on its own.

True! So it wouldn't be as hellish to write our own allocator as it
would be for most projects, because we've done 90% of the work already
(including the hard part: allocating things of variable size).

> hash-tables, and a few more; and "markers" similarly include a few
> non-marker objects, mostly overlays.  Oh, and "intervals" are objects
> used to store text-properties.

Yeah. As an aside: most of the inside of Emacs is quite nice, but I am
seriously unimpressed by overlay storage. Linear scans for
'overlays-at'? Ew. One of these years I might try to implement something
better, like what XEmacs did for extents, so you can find extents over a
given buffer position more efficiently... (not because this is visibly a
problem in profiles that I've seen, merely because the inefficiency
offends me :) )

>> I suspect actually proving my contention first would be a good
>> idea.  Not sure how to get the addresses of Lisp objects from a running
>> Emacs though: gdb, presumably.
>
> I'd hack src/alloc.c to export the needed info to Elisp.  But maybe
> memory-usage will already give us enough info that this won't
> be necessary.

Already done by Dmitry before I could do more than blink :)

-- 
NULL && (void)