Re: trunk r114534: Get Ruby's SMIE code to pass the test suite.

[Dmitry Gutov]

On 21.10.2013 20:46, Stefan Monnier wrote:
> > > Maybe change the tokenizer so that a ".qux" *at smie-bolp* is tokenized
> > > as "." and "qux"?
> > That will probably break the following modified example:
> > class C
> >    def foo
> >      self
> >        .end
> >      D.new
> >       .class
> >    end
> > end
>
> Yes, the problem is that the set of desired indentation does not match
> the structure of a fixed parsing.  So, some of the differences need to
> be handled in ad-hoc ways in the ruby-smie-rules function.
>
> E.g. for (:before . ".") we'd have to look at the previous token and if
> it's of the form "foo.bar", then manually align with the "." of
> "foo.bar".

Indentation is not the only problem there. Tokenizing ".end" as [".", 
"end"] also breaks sexp movement and (for example) 
`smie--matching-block-data'. The dot concatenation logic is the result 
of you fixing that in revision 114545.

> > But I guess we can make the tokenizer both return "." as a separate
> > token and include it in the token that goes after it.
>
> I'm not sure I understand exactly what you're suggesting, but I'd
> venture to say that you can't do that.

I don't see why not. I've installed a patch along these lines in 114738, 
and it fixes the existing examples. Unfortunately, it broke the more 
complex situations:

foo do
  bar
    .tee
end

def bar
  foo
    .baz
end

...apparently because (goto-char (cadr (smie-indent--parent))), when 
called before "." in these examples, brings point to the end of the 
first line, and so (smie-rule-parent ruby-indent-level) returns a 
surprising result.

Any suggestions what to do about that?