Trouble with lexical-binding.

[Alan Mackenzie]

Hello, Emacs.

I recently tried out M-x auto-insert in a file.el, and it caused a
testing macro to fail.  The critical point was the default setting of
lexical-binding to t in the first line.

Reduced to its essentials, my problem is a file looking like this:
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;;; lexical-bug.el --- lexical bug                   -*- lexical-binding: t; -*-
(eval-when-compile
  (defmacro test-ptr (x)
    `(let* ((ptr (copy-tree ,x))
            (form '(setcar ptr 'a))
            (result (eval form)))
       (message "result is %s, ptr is %s" result ptr))))

(test-ptr '(x y z))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

When I byte-compile it, I expect on loading it to get the result message
in the message area.  This works fine without the lexical binding.  With
the LB, instead I get an error about `ptr' being unbound - clearly, in
the form `(eval form)', i.e. `(eval `(setcar ptr 'a))', there is no `ptr'
in `eval''s stack frame.

Obviously, this bit of the code needs dynamic binding.  So I should be
able to bind `lexical-binding' to nil at some strategic place to achieve
this.  I'm not quite sure where this place is, but nowhere seems to
work.  For example, if I change the file to the following:
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;;; lexical-bug.el --- lexical bug                   -*- lexical-binding: t; -*-
(setq lexical-binding nil) ; 1
(eval-when-compile
  (let (lexical-binding) ; 2
    (defmacro test-ptr (x)
      (let (lexical-binding ; 3
            )
        `(let* (lexical-binding ; 4
                (ptr (copy-tree ,x))
                (form '(setcar ptr 'a))
                (result (eval form)))
           (message "result is %s, ptr is %s" result ptr))))))

(eval-when-compile (setq lexical-binding nil)) ; 5
(setq lexical-binding nil) ; 6
(test-ptr '(x y z))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
, I still get an error with `ptr' being unbound.

What is going on here?  Why is the byte compiler continuing to use
lexical binding despite the variable being set to nil in six different
ways?

What do I have to do to get `ptr' in this context a special variable
(whilst still having it lexically bound in other code)?

-- 
Alan Mackenzie (Nuremberg, Germany).