Re: Is add-to-list supposed to work when lexical-binding is t?

[Kelly Dean]

Stefan Monnier wrote:
>In order to do what you'd want them to do, they'd have to look at the
>value of variables in the scope of their caller.  This is exactly what
>dynamic scoping provides, and is what lexical scoping prevents.

But unconditionally preventing it with lexical scoping is the problem. If 
lexical quoting weren't useful, then why would C and your new gv-ref/deref 
enable it?

>   (let ((funs (mapcar (lambda (x) (lambda (y) (+ x y)) '(1 2 3))))
>     (mapcar (lambda (f) (funcall f 3)) funs))
>
>this should return (4 5 6).  Between the two mapcars, we have stored in
>`funs' 3 functions (more specifically closures), each one of them of the
>kind (lambda (y) (+ x y)).  They each have their own copy of `x', so all
>three copies of `x' exist at the same time in `funs'.

I was sloppy in my description. When eval sees the symbol x, and is going to 
interpret it as a lexical variable, as it does in your example, it looks up, in 
the current lexical environment, the address p_x of the memory cell that's the 
current instance of x, then reads from the cell at p_x the address of the 
object that's the current Lisp-level value of x, or writes some different 
object address into the cell at p_x if you do (setq x ...). So of course, at 
this point, eval knows what p_x is.

If you do
(defun foo-incf (x y) (setf (gv-deref x) (+ (gv-deref x) y)))
(let ((funs (mapcar (lambda (x)
                      (lambda (y) (foo-incf (gv-ref x) y) (* x 2)))
                    '(1 2 3))))
  (mapcar (lambda (f) (funcall f 3)) funs)) -> (8 10 12)
then in each of the 3 closures in the list funs, there's a call to foo-incf, 
with the first argument being a cons cell (with a pair of closures) returned by 
the expanded form of (gv-ref x).
What I'm proposing is that, instead of using (gv-ref x), have the special form 
(quote-lex x) return p_x, and use this as the first argument to foo-incf. Each 
of the 3 times the function (lambda (y) (foo-incf (quote-lex x) y) (* x 2)) is 
called, there will be a different current environment with its own instance of 
x, but eval knows which environment is current, so it does know what the 
correct p_x is, so it knows what to evaluate (quote-lex x) to.
p_x is then passed to foo-incf, which can use symbol-value (modified to accept 
p_x, i.e. a lexical instance reference) instead of using gv-deref. So foo-incf, 
add-to-list, etc will work regardless of whether lexical binding is enabled.

>We can definitely make add-to-list work for
>
>   (let ((x '(a))) (add-to-list (gv-ref x) 'b) x)   ===>   (b a)
>
>That's easy and would work fine.  But (gv-ref x) is not the same as 'x

Indeed not the same: I get "Lisp error: (wrong-type-argument symbolp ((closure 
..." because gv-ref returns a pair of closures that just imitate a lexical 
instance reference, so to make it work, I have to modify add-to-list and 
replace all the calls to symbol-value by calls to gv-deref, and change "set 
list-var" to "setf (gv-deref list-var)"; this solution is essentially the same 
as the add-to-list-lexable solution using wrap-/get-/set-passed-lexical in my 
original message, so it has the same problems: it's incompatible with current 
use of add-to-list (857 occurrences in Emacs 24.3 el and texi files) and any 
other functions that use set and symbol-value on an argument, it requires 
lexical binding, and it's inefficient, with the inefficiency causing macros 
instead of functions to be necessary as a workaround. Even if you modify set 
and symbol-value to accept the output of gv-ref, they can't catch type errors; 
they can distinguish a symbol from a non-symbol
 but can't distinguish the output of gv-ref from other structures of the same 
form. Real lexical quoting would solve all those problems.

(eq (quote x) (quote x)) -> t (equal addresses)
(equal (gv-ref x) (gv-ref x)) -> t (equal pairs of closures)
(eq (gv-ref x) (gv-ref x)) -> nil (but separate pairs of closures)
In contrast, "&x == &x" is true in C, even for lexical x.
C gives you true equality. Lisp only gives you separate-but-equal. My proposal 
is:
(eq (quote-lex x) (quote-lex x)) -> t (equal addresses, for either global or 
lexical x)
quote-lex wouldn't have to make a cons cell and return a pair of closures, and 
dereferencing one pointer is more efficient than the car and funcall that 
gv-deref has to do.

Here's another perspective: you pointed out in your original reply, "a variable 
is not the same thing as a symbol." In
(letrec ((mylen (lambda (x) (if x (1+ (funcall mylen (cdr x))) 0)))
         (x '(a b x)))
  (funcall mylen x))
the symbol x occurs 5 times as 2 different variables (the first bound to 4 
different instances at runtime, the second bound to 1) and occurs once not as a 
variable.
When you do (setq cursor-type 'bar), bar is just a symbol, so it's correct to 
use (quote bar) to get that symbol. But when you do (setq indent-line-function 
'my-indent-func), my-indent-func is a symbol that's interpreted by funcall in 
indent-according-to-mode and indent-for-tab-command as a variable (ignoring the 
Lisp-2 issue), so using (quote my-indent-func), which just returns a symbol, is 
a pretense that a symbol _is_ the same thing as a variable. That's a type 
error, but Lisp can't catch it. Since you want the global variable, and there's 
only one that the symbol can be interpreted as, the pretense isn't a problem. 
But when you do
(let ((x '(a))) (add-to-list 'x 'b) x)
it fails because the first parameter to add-to-list is a variable, but you pass 
just the symbol x, and this time the type error bites you, since add-to-list 
interprets the symbol x as the wrong variable. So the problem isn't really lack 
of _lexical_ variable quoting; the problem is that in Lisp, you can't quote 
variables at all. You can only quote symbols, and pretend that symbols and 
variables are the same thing. So quote-lex would more appropriately be called 
"quote-var". Sorry for the name change; obviously I didn't think this through 
enough.
A lexical instance reference is in effect a symbol plus a reference to the 
environment in which to interpret that symbol as a variable, so modifying 
symbol-value to accept such references just makes it stop assuming that you 
always mean the global environment.

For completeness, either (setq indent-line-function 'my-indent-func) should 
signal a type error, or "'x" should be read as (quote-var x), and (setq 
cursor-type 'bar) should signal a type error (so use (setq cursor-type :bar) 
instead), but this would break everything. But just adding quote-var, and 
continuing to allow symbol<->variable conflation for global variables, wouldn't 
break anything.

>and trying to magically turn one into the other, while feasible in a few
>particular cases is simply impossible in general.

I don't see cases where quote-var would fail to do what's intended.

>Argument-mutating functions are relatively rare and using macros tends
>to be a lot more efficient than using gv-ref, so in most cases using
>macros makes more sense, despite their well known disadvantages.

If they're rare then I guess my proposal is pointless. But for the ones that do 
exist, I think quote-var would be suitable.

BTW sorry about my mis-threaded replies, but I read from 
lists.gnu.org/archive/html/ (since I don't subscribe to lists) and it omits 
Message-ID: headers, so I have nothing to put in a References: header.