bug#26977: 25.2.50; RNC mode indentation is bad

[Ivan Andrus]

Using rnc-mode from ELPA (version 0.1), I noticed a problem with
indentation stemming from improper tokenization.  In particular
smie-default-forward-token returns all contiguous punctuation characters
as a single token.  Thus the RNC snippet below indents as


default namespace = ""

start =
    element test {
        attribute client-type { text }?,
                              attribute client-version { xsd:NMTOKEN },
        attribute trusted-key { text }?
    }


but if a space is added after the ? and before the , then it indents
(properly) as

default namespace = ""

start =
    element test {
        attribute client-type { text }? ,
        attribute client-version { xsd:NMTOKEN },
        attribute trusted-key { text }?
    }


The patch below fixes it for me, but there is likely a better way to do
it.  It relies on the fact that no token in RNC is more than one
punctation character which seems somewhat coincidental, but is true
as far as I know.

I'm happy to push the change if it's acceptable (I guess I would have to
update the version as well), or someone else can.

-Ivan


diff --git a/packages/rnc-mode/rnc-mode.el b/packages/rnc-mode/rnc-mode.el
index 9a08b5ebb..cdfeec34e 100644
--- a/packages/rnc-mode/rnc-mode.el
+++ b/packages/rnc-mode/rnc-mode.el
@@ -106,7 +106,12 @@
                (forward-comment -1)
                (= (point) start)))
         " ; "
-      (smie-default-forward-token))))
+      (if (looking-at "\\s.")
+         (buffer-substring-no-properties
+          (point)
+          (progn (forward-char 1)
+                 (point)))
+       (smie-default-forward-token)))))
 
(defun rnc-smie-backward-token ()
   (let ((start (point)))
@@ -118,7 +123,12 @@
                    (looking-at "\\(?:\\s_\\|\\sw\\)+[ \t\n]*[|&]?=")
                  (goto-char pos))))
         " ; "
-      (smie-default-backward-token))))
+      (if (looking-back "\\s.")
+         (buffer-substring-no-properties
+          (point)
+          (progn (forward-char -1)
+                 (point)))
+       (smie-default-backward-token)))))
 
(defun rnc-smie-rules (kind token)
   (pcase (cons kind token)






In GNU Emacs 25.2.50.2 (x86_64-apple-darwin15.6.0, NS appkit-1404.47 Version 
10.11.6 (Build 15G1421))
of 2017-05-17 built on iandrus-osx
Repository revision: 38880d12736909ce7ea0631562af858a1de05e0e
Windowing system distributor 'Apple', version 10.3.1404
Configured using:
'configure --without-makeinfo PKG_CONFIG_PATH=/opt/X11/lib/pkgconfig'

Configured features:
JPEG RSVG DBUS NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix