Re: Thoughts on the buffer positions in the byte compiler's warning messages.

[Helmut Eller]

On Sun, Sep 18 2016, Alan Mackenzie wrote:

> After the reader has produced the form to be compiled and
> read-symbol-positions-list, we combine these to produce a @dfn{shadow
> form} with the same shape as the form, but where there's a symbol in the
> form, there is a corresponding list in the shadow form, noting the
> corresponding "position" in the form, and onto which warning/error
> messages can be pushed.  These can then be output at the end of the
> compilation.

It would be problematic to pass such shadow-forms to macros because
macros would be confused if they see artificial lists where they expect
symbols.  After macro expansion that's no longer a problem and the
compiler can use any representation that is convenient.

So, instead of replacing symbols with lists, I would attach the position
to cons cells (stored in an auxiliary hashtable with the cons as key).

(Remember that in Lisp code almost every symbol is stored in the car of
of a cons.)

> Comments?

First, I would look at CMUCL or Clozure CL for inspiration.  Those
compilers produce fairly accurate source location information and also
have the problem that they need to pass "vanilla" forms to macros.
(Scheme compilers use "syntax-objects" instead of vanilla forms, so
there's probably not much to learn there.)

Second, instead of (or in addition to) recording the position of symbols
in read-symbol-positions-list I would record the start and end position
of lists ie. the READ function should record the position of "(" and the
corresponding ")" for each list.

Third, MACROEXPAND should record source forms so that the compiler can
list all forms (and positions) that eventually generated the fully
expanded form.

Fourth, for very sophisticated macros there should probably be some API
beyond MACROEXPAND so that macros can help the compiler to track source
positions.

Helmut