[bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Scott McPeak]

Follow-up Comment #3, bug #18396 (project make):

Regarding efficiency:

First, the main point of my report is not "alloca sucks" but rather
that setrlimit is an unexpected thing for make to do.

But having opened up the can of worms, let me play with them a little
(I don't really want to start a big argument, but I do want to defend
my previous statements):

In my experience, the dominant effect on program performance, after
choice of algorithm, is data locality: frequently re-using the same or
nearby memory locations.  Instruction count is at best a second-order
effect.  Andrew Appel has a lovely paper where he "proves" that (of
all things) garbage collection is faster than stack allocation, by
exclusively focusing on instruction count:

  http://citeseer.ist.psu.edu/appel87garbage.html
  
Of course, the analysis is fatally flawed by ignoring locality.

My point regarding alloca is that if I have a lot of data, it cannot
all be "hot" (high locality).  If I put cold data on the stack, the
stack as a whole has poorer locality, because the hot regions (stack
frames) are interspersed with cold regions (the big chunks of data).
So if I intend to allocate infrequently-accessed data, it is better to
put it on the heap where it won't interfere with the locality of stack
accesses.

You say that only "medium" sized data ends up on the stack, which
certainly sounds like a reasonable approach, but if there is enough of
it to warrant calling setrlimit, then I have to suspect that some of
it would be better placed elsewhere, from a strictly
performance-oriented point of view.

Regarding portability:

Indeed, there is no portable way to detect running out of stack space,
whether by alloca or not, though alloca obviously makes running out of
stack space more likely.  Of course, setrlimit is not all that
portable either, so I was assuming that nonportable solutions were on
the table.  Toward that end, an appropriate signal handler (which will
need to use nonportable tricks to distinguish stack overflow from
other types of segfaults), or the GCC -fstack-check argument (when
compiling with GCC, which is after all the common case) may be options.

Moreover, I would argue that the architectural decision to make heavy
reliance on alloca is what leads us into nonportable territory in the
first place.  It is not uncommon to find systems with ~10MB or less of
stack space.  The setrlimit call is a band-aid; in some circumstances
it will help, but that call is not an antidote to running out of stack
space.

Of course, the big advantage of alloca is that it is easy to program
with.  I would not advocate suddenly rewriting GNU make to use malloc
everywhere, since (as you say) that will likely introduce many more
serious bugs than this one in the short term.  (If it was written in
C++ there would be better options, but that ship too has sailed.)

So, at most, I would suggest doing some profiling to find the biggest
consumers of alloca space and change just them (hopefully only a
handful) to use [x]malloc instead.  The primary motivation would be
enhanced portability; the efficiency argument is just meant to allay
fears that this will make everything slower.

Regarding setrlimit itself:

Yes, I would consider this bug (and I do consider it a bug...) to be
fixed if 'make' would reliably return the resource limits to their
original settings before exec'ing its children.


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?18396>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/