Problems with debug-on-entry in the Lisp debugger.

[Lute Kamstra]

Debug-on-entry currently changes the definition of functions by adding
code that enters the debugger.  This code needs be hidden from the
user of the debugger, which can be hard.  Redefining the function also
removes this code and thus cancels debug-on-entry, which is
inconvenient.  I'll give some more details on these problems below and
propose an new implementation of debug-on-entry that (hopefully)
solves these problems.

I'm currently working on three bugs in the debugger that have to do
with debug-on-entry:

1. When using `d' in the debugger to step through code, you can
   encounter the code added by debug-on-entry.  For example: you have
   two functions, f-outer and f-inner, and f-outer calls f-inner, and
   you set both to debug-on-entry.  If a call to f-outer puts you in
   the debugger and you decide to step through the function with `d',
   then you will encounter the code added by debug-on-entry.  This is
   not only bad because it shows the debugger's internals and can thus
   cause confusion, but it also bad because it effectively stops you
   from stepping into f-inner since you will just step into the
   debugger.

   The only solution (within the current implementation) that I can
   think of, is to temporarily remove all debug-on-entry code while
   stepping with `d'.  This could change the definition of the
   function currently being debugged, which is rather tricky.

2. The debugger keeps a list of functions, debug-function-list, that
   are set to debug-on-entry.  Redefining a function set to
   debug-on-entry effectively disables debug-on-entry but doesn't
   remove it from debug-function-list.

   Richard proposed to change defun, defmacro, defsubst or defalias to
   let them put in debug-on-entry code when a function is in
   debug-function-list.  Stefan suggested to use advice to cause
   functions to debug-on-entry.

3. Debug-on-entry for macros is currently broken: debug-on-entry just
   strips the car (i.e. `macro') of the macro definition and treats it
   as if it was a function.  This not only makes debug-on-entry for
   macros useless, but it also breaks the macro definition itself;
   even a subsequent cancel-debug-on-entry will not put `macro' back.

   I can think of two points in a macro to set a break for the
   debugger: just before macro expansion and just after it, right
   before the evaluation of the resulting sexp.  In both cases, hiding
   the debug-on-entry code from the user of the debugger seems not
   possible.

When I was thinking about these three problems, it seemed to me that
the easiest and simplest thing to do, is to move support for
debug-on-entry into the C implementation of the Lisp interpreter.  To
mark a function for debug-on-entry, you could set the debug-on-entry
property of the function's symbol and the Lisp interpreter would then
call the debugger.

This doesn't change the definition of functions so it hides the
debuggers internals much better.  So it solves problem 1.  Redefining
a function does not effect the debug-on-entry property on the symbol
so problem 2 is solved as well.  From the lisp interpreter it would
also be easy to call the debugger before and/or after macro expansion,
thus solving 3 as well.

The big disadvantage is that the Lisp interpreter has to check the
properties of every functions it evaluates.  This slows down the
overall speed of the interpreter.  This could be mostly solved by
using a `debug-on-entry' variable that defaults to nil.  The Lisp
interpreter will only consider the debug-on-entry property of
functions if the debug-on-entry variable is non-nil.  The debugger
would then only set this variable to a non-nil value if the
debug-function-list is not empty.  So the overhead of moving
debug-on-entry support into the Lisp interpreter would then be one
variable check per function call.  This seems acceptable to me.

Shall I go ahead and try to implement this, or do people think this is
a bad idea?

Lute.