bug#23529: Request for fixing randomize_va_space build issues

[Eli Zaretskii]

> Cc: p.stephani2@gmail.com, philippe.vaucher@gmail.com, 23529@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Fri, 9 Sep 2016 12:59:16 -0700
> 
> On 09/09/2016 11:45 AM, Eli Zaretskii wrote:
> > All of those data structures are memory allocated for Lisp objects and
> > their supporting structures, with known structures, so we know exactly
> > which pointers need fixing.
> 
> Of course. But it's not trivial to fix them. It can be done, but it will 
> take code that will be hard to maintain portably.

I fail to see why it would be hard to maintain that portably.  Those
data structures are entirely our design and implementation, their
differences between platforms are almost non-existent.  Finding all
the pointers in them is almost trivial.

> > gmalloc is already implemented
> 
> Yes, and its problems are prompting this discussion. gmalloc was a fine 
> design for the 1980s but is not now.

temacs is not a program that needs to run for prolonged time
intervals, its only purpose is to produce the data that the un-dumped
Emacs will use.  So whether its malloc implementation is strong enough
by today's standards is not a relevant question.  What matters is is
it good enough for what temacs should do before it exits.

> > If there are libc's out there that allow the application to define its
> > own sbrk, then we could use that (we do on Windows).
> 
> The sbrk model is becoming less and less plausible.

Or whatever other back-end is used by malloc implementations, sbrk is
not an important detail.

> > If not, gmalloc
> > will be good enough for the temacs run; emacs will of course use the
> > normal libc allocators.
> 
> This would give up on redumping, no?

Not necessarily, we could have a variable that would force using the
pre-dump malloc in emacs.

> Plus, it assumes sbrk, which is backward-looking.

What part assumes sbrk?

> POSIX has withdrawn support for sbrk and there is 
> movement to deprecate it in C libraries due to security/robustness 
> concerns. This is something we should encourage, not run away from.

This is a wrong tree to bark up.  What we need is a malloc back-end
that will allow to allocate memory off an implementation-specified
memory block, that's all.

If we cannot have that (which would surprise me, since MS-Windows does
provide such a feature), we can still implement undump using a data
file, but it will make our job slightly more complex, as we'd need to
collect the data allocated off the heap before dumping it.  Not rocket
science, either.

> > What is a "block" in this context? Surely, a data structure with
> > linked pointers cannot be distributed between different "blocks",
> > since a linker will not know how to fixup each address, because it
> > doesn't understand the data structure.
> 
> It can be distributed between different "blocks", because we can tell 
> the compiler and linker the data structure. Here's a quick example with 
> two small "blocks" dX and dY (the actual code would differ, this is just 
> a proof of concept):
> 
>    /* Simplified version of lisp.h.  */
>    #include <stdint.h>
>    typedef intptr_t Lisp_Object;
>    enum { Lisp_Int0 = 2, Lisp_Cons = 3 /* ... */};
>    #define make_number(n) (((n) << 2) + Lisp_Int0)
>    #define TAG_PTR(tag, ptr) ((intptr_t) (ptr) + (tag))
>    #define Qnil ((Lisp_Object) 0)
>    struct Lisp_Cons { Lisp_Object car, cdr; };
> 
>    /* Define a statically-allocated pair x that is equal to (10).  */
>    struct Lisp_Cons dX = { make_number (10), Qnil };
>    #define x TAG_PTR (Lisp_Cons, &dX)
> 
>    /* Use x to build a statically-allocated list y that is equal to (5 
> 10).  */
>    struct Lisp_Cons dY = { make_number (5), x };
>    #define y TAG_PTR (Lisp_Cons, &dY)

But we don't do these things in our code, so how is this relevant to
this discussion?

What I had in mind is the data structures we use to support
maintenance of Lisp objects.  One example is string_blocks, which we
use to maintain Lisp strings.  Surely, this structure will be in a
single "block" under memory randomization, right?

> > We won't be able to use them as just compilers and linkers. We will
> > be using them for a job that is quite a bit more complex and
> > different.
> 
> No, this sort of thing is something that compilers and linkers do all 
> the time.

We won't know for sure until this is fully implemented.

Anyway, my take from this discussion is that we shouldn't give up so
easily on dumping data as a binary file, as that approach sounds to me
more future-proof than relying (again) on external technologies that
were not meant for this specific job.