Re: Dumper problems and a possible solutions

[Rich Felker]

On Tue, Jun 24, 2014 at 03:27:39PM -0400, Stefan Monnier wrote:
> > To solve ALL of the problems with the dumper (which seems to be a
> > recurring theme), I have a proposed design to make it fully portable
> > -- even moreso than xemacs "portable dumper" which is still an ugly
> > hack. The idea is simple: after loading all of the lisp objects that
> > need dumping, walk the lisp heap and output a representation for each
> > object as a giant static array in C source format, then compile and
> > link this new translation unit with the rest of the emacs .o files to
> > produce a final emacs binary. No hacks with binary formats would be
> > involved; everything would happen at the C source level. As part of
> > the lisp heap dumping, address references to other objects would have
> > to be relocated to refer to the object's position in the static array
> > rather than the original address at which the object resided when
> > created in temacs. That's some non-trivial work, but definitely no
> > prohibitive, and as a bonus, the generated address-constant references
> > in the static array would transform to load-address-relative
> > relocations for the linker, allowing emacs to be built as a
> > position-indepdendent executable (PIE) if desired.
> 
> Generating a big static C array against which to link sounds fine and
> very portable, indeed.  I'm not sure how hard/easy the relocation could
> turn out to be.  There's the problem of finding *all* the references,
> and there's the problem that moving an object means that its "hash"
> value changes.

Thanks for the feedback. Can you elaborate on how/why the hash
changes, and where it's stored that would need to be updated? As far
as the relocation, my impression is that it would just need to be able
to identify pointers in lisp objects (this is already possible since
the GC needs to do it, right?), and rewrite them to (essentially)
"static_lisp_heap + offset_of_pointed_to_object" when writing the dump
out as a C array.

> > If not, or if that's going to be a very long-term project, would a
> > cleaned-up version of my current solution be acceptable upstream?
> 
> Making the "dump" portable would be very welcome.  Generating a big
> static C array sounds OK.  So whether the result is acceptable or not
> will depend on what's needed to solve the problems linked to relocation.
> 
> Another option is to "dump" the heap into a binary file that we would
> later on "mmap".

This is the xemacs "portable dumper" approach, and I believe it's
inferrior because it depends on being able to map back at the same
location. If the region is a page-aligned static buffer in the main
executable and you mmap over it with MAP_FIXED, this is safe for the
most part, but it's still incompatible with PIE. I think it would be
nice to solve this problem in a way that also makes PIE possible.

Rich