Re: CANNOT_DUMP support

[Stephen J. Turnbull]

Richard Stallman writes:

 >     XEmacs has not had an unexec issue since 2001, because of the
 >     introduction of the "portable dumper".
 > 
 > It sounds interesting.  
 > 
 >     The portable dumper transparently ignored the "Hannibal Lector"
 >     ld and (mostly) the "Chaos Incarnate" SELinux loader.  It also
 >     has made preload support for new platforms trivial.

The "Hannibal Lector" ld was an innovation that optimized the order of
ELF sections (?) in X?Emacs's executable, causing the unexec
computations to be incorrect (the end of code marker wasn't beginning
of data or something like that).  The SELinux loader randomly changes
the position of the programs sections at load time to make buffer
overrun exploits harder (or something like that, I'm not an SELinux
geek).  I'm not sure if we've addressed that yet, I think the portable
dumper did fail with that setting in SELinux.  ISTR that Olivier said
it could be handled though.

 > Can you tell us more about how this works and what it does?

The basic idea is that we load up Emacs with its Lisp, then start with
the GC roots and "wire up" (for technical details, ask Olivier) the
objects referenced with offsets, then write them to a file.  At
runtime, loading the file does the reverse.  I'm not sure how this is
done, whether the portable dumper actually traces all the offsets and
converts them to appropriate pointers at runtime, or whether the
system loader does this as part of its normal ELF relocation link
editing.  Whatever, it's not as fast as unexec, but it's real fast.

Of course you want to put the "file" into the emacs binary.  This has
been done but there were some minor issues.  I don't recall whether it
was lack of time by those who know about the feature or a real problem
that was solved.

 >     may even be legally[1] usable, as the number of contributors is
 >     very small and all (except maybe Kyle Jones) would probably
 >     sign assignments.
 > 
 > I don't think we even have contact with him any more.
 > Do you?

Not directly, but the VM maintainers do.

 > Supposing he won't sign, how hard is it to rip out and replace his
 > code?

You'll have to ask Olivier about issues that have to do with what is
where in the code.

 >       (It does depend on the layout and types of builtin Lisp
 >     objects, so probably it wouldn't help all that much, though.)
 > 
 > Do you mean that it would be hard to adapt it for the differences
 > in those layouts?

The portable dumper and runtime loader itself is a fairly small amount
of code.  Instead, each class of Lisp object gets its own marshalling
routines because of the various ways that data is incorporated.  Eg,
conses are trivial to save but you have to follow car and cdr.
Strings are a mess because they refer both to Lisp objects (any text
properties) and the string storage itself is not contiguous to the
string object and has to be described ad hoc.  And so on and so forth.

I don't think it's "hard", just tedious.

 > Could we adapt the idea, at least?

Yes; it's inherent in the pointer discipline in the Emacs Lisp
implementation, which (at this level) we share.  I'm sure it's been
proposed for Emacs, too.  What I'm contributing here more than
anything is (a) yes, it works, we've done it, and (b) a pointer to
Olivier who can tell you how much work is involved and probably will
consult on implementation (depending on his personal time
constraints).