Re: Elisp development news

[Neil Jerram]

>>>>> "Marius" == Marius Vollmer <address@hidden> writes:

    Marius> Neil Jerram <address@hidden> writes:
    >> So, would anyone else object to me
    >> 
    >> - deprecating the unnecessary C Elisp support in the stable
    >> branch
    >> 
    >> - removing the same stuff in the CVS head?

    Marius> Hmm, my little finger tells me to leave this stuff in for
    Marius> now.

    Marius> What is your plan re data sharing between Elisp and Guile
    Marius> Scheme?  What about Jim's proposal?  Etc.  Until we have a
    Marius> firm grasp of these things and are sure we are not acting
    Marius> out of ignorance of what has been designed previously, we
    Marius> shouldn't start changing stuff.

OK; on the main point - Elisp and Scheme interaction - I am very happy
to have this discussion before we start changing stuff.  Otherwise I
too would be quite worried that I have missed something!  To make a
start on this discussion, I'll add Jim's proposal below and comment on
where my design differs from it.

But how about 1-ify, 0-ify, 0-cond?  Do we really want to keep these?
Does the ctax code use them?

Now here's Jim's proposal and my comments on it...

  From: Jim Blandy (address@hidden)

  Here's a plan for allowing Emacs Lisp and R4RS Scheme to share data
  structures in Guile with pretty good transparency.

  First, some background:

  Guile will support both Emacs Lisp and Scheme, and allow code written
  in one language to exchange values and share data structures with the
  other. Because Emacs Lisp and Scheme have very similar type systems,
  this sharing can be almost transparent.

This is very important; it's only because Lisp and Scheme are so
close already that transparent data sharing is attractive and all
the following issues arise.

Given that Elisp/Scheme data sharing is so attractive, we
especially want to avoid any approach that would require
translating data as it passes between the two worlds.  (Think about
the performance implications for arbitrarily complex structures!)

  However, there are differences. Emacs Lisp, following lisp tradition,
  uses the symbol `nil' to represent the empty list and Boolean false.
  On the other hand, Scheme has distinct objects for all three uses: the
  symbol `nil' has no special significance, the empty list is written
  '(), and the special object #f is false. (It's the only false object,
  in fact; all other objects are true in Scheme, including the empty
  list.)

Note: in current Guile CVS, in Scheme, `nil' and `t' are bound to
built-in variables whose values are the symbols `nil' and `t'.  My
implementation removes these (in Scheme).

  We would very much like to find an arrangement that allows all Emacs
  Lisp code to run unchanged, allows all R4RS code to run unchanged, and
  allows the two to communicate mostly transparently.

Absolutely.

  None of the obvious compromises work nicely. Modifying either
  language to match the other breaks existing code in ways that are
  impossible to fix mechanically. Making Emacs Lisp's nil identical to
  either of Scheme's #f or '() will cause misinterpretations when Scheme
  receives either Boolean values or lists from Emacs Lisp.

How?  Jim isn't clear about these misinterpretations.

Given <the set of objects that Elisp code treats as false>, and an
Elisp function that may return any of these to indicate false, a
Scheme function examining this return code would have to check for all
possible Elisp false values.  I don't see how this interacts with the
question of whether nil can be identical to either #f or '().

We will get into this question in greater detail below...

  It's worth noting that this arises as a crucial issue only because
  Emacs Lisp and Scheme values are so tantalizingly similar. When
  mixing languages with more blatantly different value worlds --- C and
  Scheme, for example --- programmers should expect to be responsible
  for making the necessary conversions. But because the Scheme/Emacs
  Lisp boundary is so transparent already, it's worthwhile to clear up
  the few remaining discrepancies as much as possible.

Oops - didn't I just say that? :-)

  In the following paragraphs, I'll try to avoid confusion by using this
  notation:

  I'll write #f for the value assigned by Scheme to the expression #f.
  I'll write '() for the value assigned by Scheme to the expression '().
  I'll write nil for the value assigned by Emacs Lisp to the expressions
  nil, '(), and ().

  Guile will support Emacs Lisp by translating it into Scheme, and that
  translation need not map Emacs Lisp forms onto their most obvious
  Scheme counterparts; this gives us some room to maneuver. For
  example, Emacs Lisp's `if' form could be translated into a Scheme form
  `lisp-if', which could handle Boolean values differently from the
  normal Scheme `if'.

Agreed.  In my implementation, (if ...) is translated to a `nil-cond'
form which checks whether its predicates are (or (null? x) (not x)).

  We want list-valued Scheme and Emacs Lisp functions to interoperate.

Yes, but ...

  So Scheme must be able to return '() to Emacs Lisp, and have Emacs
  Lisp recognize it as the empty list. Emacs Lisp must be able to
  return nil to Scheme, and have Scheme recognize it as the empty list.

... not quite.

For any value X that language A passes to language B, the requirement
is that B understands what A means by X, to the extent possible given
B's own linguistic limitations.

Where null-like values are concerned,

- When A is Elisp and B is Scheme, "what A means by X" can only mean
  whether X is false/empty in the Elisp sense.  If the concept of
  Elisp false/empty maps to more than one distinguishable value in
  Scheme, note that

  - it is possible and straightforward to write a Scheme predicate
    `elisp-false/empty?' that returns #t if its argument is any of
    those values

  - it doesn't make sense for the Scheme side to try to draw any
    conclusion by examining which of those values is the one actually
    returned, since the distinction was not observable in Elisp.

- When A is Scheme and B is Elisp, "B's own linguistic limitations"
  mean that Elisp cannot distinguish between the various Scheme values
  that map to false/empty in Elisp.

So I would say that Elisp should be able to call a Scheme function
whose return value is either a non-empty list or a empty list, and
tell the difference; and that Scheme should be able to call an Elisp
function whose return value is either a non-empty list or a
false/empty value, and tell the difference.

  We want Boolean-valued Scheme and Emacs Lisp functions to
  interoperate. So Scheme must be able to return #f to Emacs Lisp, and
  have Emacs Lisp recognize it as false. And Emacs Lisp must be able to
  return nil to Scheme, and have Scheme recognize it as false.

Similarly, Elisp should be able to call a Scheme function whose return
value is either a non-false value (excepting '()) or a false value,
and tell the difference; and Scheme should be able to call an Elisp
function whose return value is either a non-false/empty value or a
false/empty value, and tell the difference.

My statements are weaker than Jim's.  For example, his

  "have Scheme recognize it as the empty list"

becomes my

  "have Scheme tell the difference between a non-empty list and a
   false/empty value".

But I think that my statements are all that is necessary in practice.

The crucial consequence of this weakening is that the set of Guile
values that describe end-of-list in Elisp do not all have to satisfy
(null? x) in Scheme, and the set of Guile values that describe false
in Elisp do not all have to satisfy (not x) in Scheme.

This opens the door for proposing that `nil' be made identical to #f
and Elisp's '() the same as Scheme's '().  All the requirements to do
with Elisp's inability to distinguish nil and '() can then be handled
in the handful of functions and macros that manipulate these values.

What about the exception that I sneaked in a few paragraphs back?
According to my weaker requirements, a boolean-valued Scheme function
that returned a empty list (which is _true_ in Scheme) to Elisp would
have its return value _misunderstood_ by Elisp.

In my view this is simply poor interface design.  If a function is
designed to be called by Elisp, its possible return value must be
presented in a way that is distinguishable by Elisp.  In practice,
given an existing Scheme function like this that you want to interface
to Elisp, it is trivial to write a Scheme wrapper that makes the
different return values distinguishable.

[Actually, I don't think Jim's approach fares any better with this
case anyway; hence the question from Thomas Bushnell at the end of
Jim's notes.]

  Since it is standard in Emacs Lisp to test for the empty list with
  `if' directly, Emacs Lisp must also recognize '() as false.

Agreed; this is straightforward regardless of how null-like values are
implemented.

  I don't see any harm in telling Emacs Lisp that #f is an empty list.
  That merely equates two objects that Emacs Lisp never distinguished
  anyway.

I'm not clear what Jim means here.  If that (if (null X) "yes" "no")
=> "yes", where X is the Scheme #f value, fine.

  So, according to the above paragraphs:
  - The Emacs Lisp end-of-list values are: nil, '(), #f
  - The Emacs Lisp false values are: nil, '(), #f (same as above)
  - The Scheme end-of-list values are: '(), nil
  - The Scheme false values are: #f, nil

In my implementation:
  - Elisp `nil' and `t' are read such that they are identical to `#f' and `#t'
  - Elisp end-of-list values are: #f, '()
  - Elisp false values are: #f, '()
  - Scheme end-of-list values are: '()
  - Scheme false values are: #f

  Because Emacs Lisp code frequently uses the `eq' function to examine
  list structure, Emacs Lisp's `eq' should treat nil, '(), and #f as
  identical. Scheme's eq? must be able to distinguish the three.

Agreed, except that in my case there is only #f and '().

  When not mixing Scheme and Emacs Lisp, this preserves both languages
  intact. When mixing the two, this supports the interactions discussed
  above.

  Where does this screw up?

  I'm told some people like to write Scheme functions that return either
  a (possibly empty) list, or #f to indicate some sort of 'other' case.
  If Emacs Lisp calls such a function, it won't be able to tell whether
  it got an empty list or the 'other' case. Here's another symptom of
  the same clash: if Scheme places #f in the cdr of a pair, Scheme will
  consider it an improper list, while Emacs Lisp will consider it a
  proper list.

  I don't see anything we can do to resolve this conflict, without
  simply changing some of the user's code; such Scheme code simply
  provides an unfortunate interface for Emacs Lisp callers. We think
  this is rare enough that it will suffice to tell programmers, "the
  Emacs Lisp/Scheme boundary is almost transparent, but this is one
  glitch you'll have to cope with."

Agreed.  My implementation is the same.

  Note that this approach does require Guile to distinguish between nil
  and '(); this is necessary to allow Scheme callers to accept values
  from Emacs Lisp Boolean-valued functions.

Not sure I understand this.  As I say above, what's necessary is for
the Scheme caller to determine whether or not the Elisp function
intended its return code to mean false/empty.  This is not a problem,
and I don't see what is has to do with distinguishing nil and '().

  Can this be implemented efficiently?

  I think so. `eq' can use fast, approximate tests to cover most common
  cases, and then fall back to a slower, correct function for the rest.
  Something like this might be useful in the Emacs C code:

  #define EQ(x, y) ((x) == (y) \
  || ((((x) ^ (y)) & MASK) == 0) && lisp_eq ((x), (y)))

  where MASK has very few zero bits, and the elements of the equivalence
  classes have representations differing in only those bits. The '=='
  clause will catch most cases when EQ is true; the '^' clause will
  catch most cases when EQ is false; and the lisp_eq function will be
  always correct, but relatively slow.

Agreed; this is quite important, I think, and basically the same with
my implementation.

  I think this arrangement meets the needs of most of the Guile audience.
  - Emacs Lisp programmers want their existing code to work when
  Emacs switches over to Guile. They won't notice a thing.
  - Some Scheme programmers want to use Guile as an ordinary Scheme
  interpreter, to take advantage of its debugging and systems
  programming features. They won't notice a thing.
  - Some Scheme programmers want to program Emacs in Scheme.
  There is no legacy of Scheme-for-Emacs code that we care about, so
  these people will deal with whatever we give them.
  - Programmers writing new applications using Guile should be happy,
  too. There is almost no extant Emacs Lisp code which doesn't use
  buffers and other Emacs-specific data types; any Emacs Lisp code
  they want to use in their application will need porting anyway.

This is that focus thing again, isn't it? :-)

Seriously, no issue here, although I'm not sure I understand the
thrust of the last point.

  Appendix: Why not just fix all the Emacs Lisp code?

:-)

  [Unnecessary explanation deleted]

  Subject: elisp needs to distinguish '() and #f, specially
  From: address@hidden (Thomas Bushnell, n/BSG)
  To: address@hidden
  Subject: Re: Guile status report
  Date: Tue, 10 Sep 1996 15:52:57 -0400
  X-Name-Change: My name used to be `Michael'; now it is `Thomas'.
  X-Tom-Swiftie: "I'm sorry I broke your window," Tom said painfully.

  Suppose I write a scheme function that wants to distinguish returning
  '() and #f.

  New Emacs lisp needs to be able to call such functions and know what's
  going on. I think the best way is to export Scheme's eq? to emacs as
  something like `scheme-eq' so that cognizant Lisp programs can do the
  right thing.

  Thomas

This is the bad interface design example that I discussed above.
Certainly one could export a `scheme-eq' to Elisp, but "cognizant"
Elisp programs would effectively be making assumptions about the
implementation.  (Specifically, that data is not translated at all as
it passes between Elisp and Scheme.)

                   ........................

Thanks and congratulations to anyone who's still reading!  I look
forward to hearing your thoughts.

        Neil