Re: [Bug-apl] Deep Nesting

[fred]

Juergen

Note that stack red-zones are typically already generated. So, you
don't want to mess with that. The following plan will work in
conjunction with the system stack red-zone.

What I would do is:

At startup, call down, with a defined limit. Each call level should
take the local storage that is expected during "normal" calls. If this
fails, the original target could not be achieved anyway. The signal()
handler can be introduced before this is done. If a trap occurs, less
stack nesting can be tried. On the deepest nesting, return a pointer to
a local stack variable. Of course, this pointer cannot be used after
return, but we just want to know the *location* of that memory.

Use mprotect() on that pointer to establish the red page. a SIGSEGV 
 will then be generated if the stack extends to that point. signal()
will capture the SIGSEGV. Use setjmp() and longjmp() to unwind the
stack after the SIGSEGV.

Now, you want to use sigaction() to supply a sigaction handler, which
gets passed a ucontext_t, which allows access to the actual address
that caused the fault, the registers, etc. Old POSIX definition had it
defined such that the fault could be processed, and a return from
the signal handler would restart the instruction. More recent POSIX
simply marked this as "undefined behaviour".

This will work with both grow-up and grow-down stacks, *and* on
implementations with "software stack". Eg. should work on Z-series.
However, it is *not* (and cannot be) POSIX compliant. POSIX only allows
mprotect() on mmap() memory (which is not stack memory). Also, POSIX
specifies that execution becomes undefined after SIGSEGV -- even if it
is caught. I just ignore these issues... (old POSIX, and very helpful
if writing debug routines, and binary translators etc.).

FredW