Re: lexical mumblings

[Andrew Innes]

On 27 Oct 2001 13:40:36 +0900, Miles Bader <address@hidden> said:
>Miles Bader <address@hidden> writes:
>>If `load' and `eval-region' just bound the variable `use-lexical-binding'
>>to itself (e.g., (let ((use-lexical-binding use-lexical-binding)) ...)),
>>Then (use-lexical-binding) could expand into (setq use-lexical-binding t),
>>and the interpreted version of `let' could key off that...(!)
>
>Well, obviously this doesn't solve the problem, I was confused.
>
>I would be interested to know the details of how Andrew's interpreted
>lexical version of let works -- I think the details will influence how
>any larger framework behaves.

Hi Miles,

I still don't have time yet to give a complete answer or response to
your proposals (though I intend to when I can), but I've attached a file
I used to capture my ramblings and musings on the subject over the
years.  It is not a linear document (some bits early on are newer than
bits that come later), but it does end up discussing some of the
implementation details I settled on.  I discuss or mention a number of
subtle issues that I hit upon over the years.  It finishes off with old
and new bytecode for several function examples.

In brief, my interpreter changes rely on a chain of environment blocks
(env_blocks in the source), which are alloca'd lisp vectors that contain
symbol followed by value.  There is a distinguished value that means
that symbol is declared special in a particular lexical (not not global)
scope and is specbound.  Symbols that are globally declared special (via
`proclaim' or `defvar') are identified by a bit in the symbol structure,
and are never lexically bound.

Flet et al push and pop env_blocks on the chain.  Feval, Fset et al
search the chain to DTRT.  Ffuncall et al start new chains as
appropriate, or reinstate saved chains when activating closures.
Ffunction evacutes the current env chain into the heap, because proper
analysis to determine if the closure might escape its binding frame is
too much to do in the interpreter.

I'd never thought of the possibility that Stefan mentioned to you, of
modifying the semantics of unbound variable lookup to emulate the
all-dynamic scoping behaviour of Elisp.  This would be a relatively
simple change to my interpreter patch, although I will have to think
carefully about it to see if there are hidden nasties in that approach.

Code written for lexical scoping should really only reference unbound
symbols that had been proclaimed or declared special, in which case my
interpreter changes already DTRT.  References to unbound symbols that
have not been defvar'd could reasonably be treated specially (pardon the
pun) by searching the stack of binding stacks, to find the most recently
instantiated binding, whether lexical or dynamic.  It would be a
violation of the expected "privacy" of lexical bindings though.  Also,
it would not work when intermixing compiled and interpreted code
(because the symbol names are omitted in env_blocks used by compiled
code).

AndrewI

PS. One of the disappointments I have about lexical binding (I'm talking
about the byte code level now) is that it is unlikely to yield any great
performance improvement, which has always seemed to me to be the
greatest incentive for such a deep change.  (Closures would be nice, but
aren't reason enough for such a fundamental semantic change.)  I don't
know if that is something you see as a prime benefit of the change - I
would be interested to hear the benefits you think it would bring.

The main benefit that I think would result is a reduction in memory
footprint, from not having real symbols created for all the variables
which really are local.  There should also be a small positive effect
from improved cache locality, with many (most?) var references hitting
on the stack instead of symbol value slots scattered throughout the
heap.  However, I strongly suspect that the total CPU time currently
expended interpreting bytcode is dwarfed by the redisplay engine, regexp
and other searching functions, etc.  Certain packages, like W3, which
really do a significant amount of bytecode work might see more
impressive improvement, but they would be the exceptions.

emacs-lexical.txt
Description: Text document