Eval, tail calls, (current-module), and backward compatibility

[Mark H Weaver]

Hello all,

There's a problem with Guile's `eval'.  It doesn't do proper tail
recursion as mandated by R5RS et al, and unfortunately we can't fix this
without changing its behavior in a potentially incompatible way.

The problem is that `eval' uses dynamic-wind to temporarily set
(current-module) during the dynamic extent of the expansion and
evaluation of its form.  Ideally, it should set (current-module) only
during expansion, _not_ during evaluation.

It is worthwhile to consider what (current-module) is for, and how it
should be used.  This seems to be an area of great confusion.

(current-module) should be relevant only at the beginning of
macro-expansion: before any program transformations are performed,
(current-module) is "baked" into every symbol of the top-level form.
(psyntax actually does this lazily, but the effect is the same).

After that, (current-module) should be completely irrelevant to the rest
of compilation and evaluation.  After expansion has begun, the expanded
code is in general a patchwork of code fragments from many different
modules, and thus it no longer makes sense to talk about "the current
module".

Instead, the compiler looks at the modules that were baked into the
identifier.  For example, each top-level variable reference refers to
the module that was baked into its corresponding source identifier
before macro expansion.

It sometimes makes sense for the user to (set-current-module <module>).
This can be done within an REPL for example.  It can also be done in a
compiled file within (eval-when (compile) ...), which will cause
subsequent top-level forms to be expanded within the newly changed
(current-module).  This is implicitly done by `define-module'.

Pretty much the only proper use of (current-module) is to implement
REPLs and things of that sort.  It has nothing to do with the code that
is currently running.  It doesn't even really have to do with the code
that is currently being expanded, so it's almost never the right thing
to look at within procedural macros.

* * * * *

Can we fix this?

Ideally, I think that `eval' should set (current-module) during
expansion, but _not_ during evaluation.  Then it can be properly tail
recursive.  However, some code out there might depend on the existing
behavior, so I guess we can't change this, at least not in 2.0.  Bummer.

  (BTW, `local-eval' accepts a module as its second argument, and
   conforms to the R5RS requirements of `eval', so it can serve as a
   proper replacement for Guile's broken `eval')

Similarly, (compile '<expr> #:env <module>) should set (current-module)
during expansion but _not_ during evaluation.  Then it can be properly
tail recursive and have cleaner semantics.  It might not be too late to
fix this in 2.0.

  (Note that `local-compile' also conforms to the R5RS requirements of
   `eval', so can also serve as a proper `eval' replacement)

What do other people think?

    Best,
     Mark