bug#22960: 25.0.92; smie-indent-keyword triggers the "Bumped into unknown token" error

[Dmitry Gutov]

When trying to indent the beginning of a multiline string literal.

Here's an example Ruby code (almost unmodified, from a real project):

foo_bar(
   "# coding:koi8-r
         \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1 = 42
         puts \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1")

Try to reindent the second line.  During its execution,
smie-indent-keyword calls smie-indent-forward-token from the third line
(in the middle of the string literal).  ruby-smie--forward-token returns
"" there (because why not?), and smie-indent-forward-token freaks out.

IMHO, smie-forward-token-function doesn't have to know how to handle
that context, and the fault lies in smie-indent-keyword.

Stefan, could you please take a look?


In GNU Emacs 25.0.92.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.7)
 of 2016-03-05 built on axl
Repository revision: dbb02bfadc38297c06bbd3462e53192d0ced76dc
Windowing system distributor 'The X.Org Foundation', version 11.0.11702000
System Description:     Ubuntu 15.10