Re: SMIE

[Stefan Monnier]

> Hi there. As the subject line says I’m writing for help with SMIE.

Cool!

> I am currently working on elixir-mode
> <https://github.com/elixir-lang/emacs-elixir>, having (apparently) taking
> over the mode as the latest in a line of contributors.

I'd love to include this in GNU ELPA.  Interested?

> Specifically I’m having trouble understanding the mental model for how
> tokenisation & indentation works.  For example, in this
> <https://github.com/elixir-lang/emacs-elixir/issues/18> issue, indentation
> errors seem to crop up only after separating lines of code with blank
> lines.
> I have spent, seriously, hundreds of hours trying to sort out what’s
> happening here and I am at my wits’ end.

IIUC, Elixir syntax does not treat all whitespace as "irrelevant",
contrary to the default tokenizer of SMIE.

> Does this issue ring any bells with issues you’ve dealt with in
> the past?

Yes, indeed.  Octave and sh are two other languages that use SMIE and
where some whitespace is syntactically significant.

What you need to do is to change the tokenizer so that instead of
skipping all whitespace, it turns the syntactically-significant
whitespace into a token (you can name it any way you like; in the above
languages, it turns out to be syntactically equivalent to a semi-colon,
so we call it ";").

I know absolutely nothing about Elixir or its syntax, so I can't give
you specific details, but you can look at octave.el and sh-script.el
for examples.  Feel free to email me back with more details if you need
further help.

> Final question, how is it determined if a token is a :list-intro token?

Not sure I understand the question.  The issue is for the indentation
rules, when it sees two (or more) concatenated expressions (e.g. "exp1
exp2"), should it assume that exp2 is something like an argument to the
exp1 function (and hence exp2 (and exp3, ...) should be indented like
a function argument) or are all those "expressions" just a list, where
the first is not more special than the second?
This usually depends on the context.  E.g. in a situation like

   fun x1 x2 x3 =>

x2 is not an argument passed to the function x1;  Instead x1, x2, and x3
are "siblings" and should be indented to the same level.  So to decide
how to indent x2 and x3 w.r.t x1, SMIE calls the smie-rule-function with
(:list-intro . "fun") so smie-rule-function can tell it that "fun"
introduces a *list* of "things" rather than being followed by a "normal
expression".

Does that make more sense?

> I have read the SMIE manual ten times, at least, but I’m really
> struggling.  I would truly appreciate your help.

I'm not very good at writing manuals, sorry.  But I promise to do my
best to help you get SMIE working well.  In return, I would appreciate
if you could help me improve the doc by giving, if not actual patches,
at least suggestions of how to rewrite the doc, or what to add to it
(usually, you can only make such suggestions after you finally
understand what's going on, and at the same time it's
important/necessary/useful to try and remember what it was that you
didn't understand).


        Stefan