Re: Debugging emacs memory management

[Dima Kogan]

Dima Kogan <address@hidden> writes:

> I'm running emacs snapshots from the latest git, usually as a
> long-running daemon. For a few months now I've had to restart my emacs
> session every week or so because it eats all my RAM, and cycling emacs
> is the only way to get the RAM back.

Hi. I'm resurrecting this thread. My issues have never gone away, and
I've been unhappily living with this bug for at least a year now.
Yesterday I FINALLY found a sequence of steps to reliably reproduce the
issue, so debugging became possible. Sequence:

1. emacs --daemon
2. Repeat:
   1. pop up a number of emacs clients
   2. kill clients

I do this in zsh:

  while true; do for i in `seq 10`; do timeout 5 emacsclient.emacs-snapshot -a 
'' -c & ; done; sleep 10; done

While this runs, the memory consumption can be monitored with

  while (true) { ps -h -p `pidof emacs-snapshot-lucid` -O rss; sleep 1 }

It looked like something in my .emacs.d/init.el was causing the leak,
and a bisection found it: winner-mode

Apparently winner-mode keeps a list of all active frames, and it doesn't
clean dead frames off of this list, so a daemon workflow where frames
are often created/removed causes it to keep references to data
structures that are no longer active, preventing the gc from cleaning
them up. I'm attaching a patch that fixes this in winner-mode.

I'm also attaching a plot of memory usage over time, as the clients are
opened and closed. As you can see, patching winner-mode makes a HUGE
difference.

However another observation is that even with an empty init.el this
sequence still leaks memory, albeit more slowly. It looks like a bug
also. I have some malloc tracing set up with backtrace reporting, but
there're a LOT of allocations, and I haven't figured out a good way to
sort through them yet.

Another observation is that the gc never really gives back the memory.
If I run the unpatched winner-mode for a while, then manually clear out
the dead frames from the list and (garbage-collect), the memory usage
(as reported by the OS) does not go back down. What does happen is that
the memory usage stays flat for a while when clients are
created/deleted, and starts climbing again much later. So the memory IS
freed, but emacs never gives it back to the OS.

I'll run with this for a while, and report back if I have any more
patches. Hopefully this fixes my main issue; we'll see. Any further
insight welcome.


>From 28a605a074afcde758ef0a2b1a7b5756f85b787a Mon Sep 17 00:00:00 2001
From: Dima Kogan <address@hidden>
Date: Tue, 15 Sep 2015 11:53:54 -0700
Subject: [PATCH] winner no longer holds on to dead frames

This prevents a potentially massive memory leak
---
 lisp/winner.el | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lisp/winner.el b/lisp/winner.el
index fdf6213..ecb009b 100644
--- a/lisp/winner.el
+++ b/lisp/winner.el
@@ -177,6 +177,11 @@ You may want to include buffer names such as *Help*, 
*Apropos*,
 ;; Called whenever the window configuration changes
 ;; (a `window-configuration-change-hook').
 (defun winner-change-fun ()
+
+  ;; cull dead frames
+  (setq winner-modified-list
+        (cl-remove-if-not 'frame-live-p winner-modified-list))
+
   (unless (or (memq (selected-frame) winner-modified-list)
               (/= 0 (minibuffer-depth)))
     (push (selected-frame) winner-modified-list)))
-- 
2.1.4

emacs_memory_consumption.pdf
Description: Adobe PDF document