bug#36432: 26.2; SMIE does not request forward tokens when point is at point-max

[Sam Halliday]

On 29/06/2019, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Sam Halliday <sam.halliday@gmail.com>
>> Date: Sat, 29 Jun 2019 13:14:01 +0100
>>
>> SMIE (via a `indent-for-tab-command') does not request forward tokens
>> from the lexer when point is at `point-max'.
>>
>> This might sound like a strange bug report: why should smie expect
>> there to be any tokens when it is already at point-max? The answer is:
>> virtual tokens. For example, Haskell may have many closing curly
>> brackets that live at the end of the buffer.
>
> ??? There can be noting at point-max, as that position is beyond the
> last buffer position.  Did you mean the position just before that?  Or
> am I missing something here?
>

I mean at point-max.

Consider this layout algorithm

https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/haskell-tng-layout.el

and this lexer

https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/haskell-tng-lexer.el

that can continue to produce tokens even when the point is at the very
end of the buffer.

e.g.

input https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/test/src/layout.hs
with layout 
https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/test/src/layout.hs.layout
just tokens 
https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/test/src/layout.hs.lexer

note the trailing } that exists at the very end of the file. SMIE
always misses this.