Calling ruby-smie--forward-token is a bit dangerous since that function
might itself be called from ruby-smie--forward-token. It might work,
but you'll have to think hard about why an inf-loop is not possible.
Hopefully because both `ruby-smie--forward-token' and
ruby-smie--backward-token' would only call `ruby-smie--forward-token', and
only when (> pos (point)), IOW there has to be some whitespace skipping done
between the recursive calls.
Only recursing in one direction (only forward or only backward) is
a good way to guarantee progress, indeed. But currently
ruby-smie--implicit-semi-p calls ruby-smie--backward-token, so beware.