bug#15212: 24.3.50; c++-mode doesn't support raw string literals

[Alan Mackenzie]

Hi, Ivan.

Thanks for the suggestion!  I've actually had an almost working solution
myself for just over a week.  Then I got confused with a bug in some CC
Mode "infrastructure" code.  Such is life!

The way I am fontifying these is thus:
(i) For a correctly terminated raw string, everything between the ( and )
inclusive gets string face, everything else just the default face:

            R"foo(bar)foo"
                 ^^^^^
          font-lock-string-face.

(ii) For a construct with a raw string opener, not correctly terminated,
I am putting warning face on the entire raw string opener, leaving the
rest of the string with string face, e.g.:

            R"baz(bar)foo"
            ^^^^^^
     font-lock-warning-face
                  ^^^^^^^^
              font-lock-string-face

Of course, that is subject to change if it doesn't work very well.

CC Mode doesn't actually use syntax-ppss and syntax-propertize-function,
since they don't allow enough control.  In particular, on a buffer
change, they erase all syntax-table text properties between point and end
of buffer which is wasteful; it is never necessary to erase these beyond
the next end of statement, and they are quite expensive to apply.

Anyhow, we should be able to have this implemented and the bug closed
pretty soon.

-- 
Alan Mackenzie (Nuremberg, Germany).



On Tue, May 24, 2016 at 11:12:31AM -0600, Ivan Andrus wrote:
> I've tried the following function which seems to work reasonably well
> in my limited testing.  I'm not suggesting this as the final solution,
> but I would be happy to do some revisions based on your feedback, or
> abandon it altogether if you have a better method.  Alternately, you
> can use any pieces that you find useful.

> (defun c++11-syntax-propertize-function (beg end)
>   ;; interactive for easy testing
>   (interactive (list (point-min) (point-max)))
>   (save-excursion
>     (if (not (nth 3 (syntax-ppss beg)))
>         ;; Not in a string, so in particular, not in a raw string
>         (goto-char beg)
>       ;; We have to re-propertize a raw-string, so move back to the
> beginning of it.
>       (goto-char (nth 8 (syntax-ppss beg)))
>       (skip-syntax-backward "'"))
>     ;; Look for raw strings in the area of interest
>     (while (search-forward-regexp
> "\\(\\(?:L\\|u8\\|u\\|U\\)?R\\)\"\\([^(]*\\)(" end t)
>       (let* ((full (match-string-no-properties 0))
>              (qualifier (match-string-no-properties 1))
>              (delimiter (match-string-no-properties 2))
>              (beg-beg (match-beginning 0))
>              (beg-quote (+ beg-beg (length qualifier)))
>              (beg-quote-after (1+ beg-quote)))
>         (let* ((ppss (syntax-ppss beg-beg))
>                (in-string-or-comment (or (nth 3 ppss) (nth 4 ppss))))
>           (if in-string-or-comment
>               ;; Move past the match to avoid an infinite loop
>               (goto-char (match-end 0))
>             ;; Search for the end of the string
>             (when (search-forward-regexp
>                    (concat ")" delimiter "\"")
>                    ;; I don't limit it to end because I'm afraid it
> might not be far enough.
>                    nil t)
>               (let ((end-end (match-end 0)))
>                 (remove-text-properties beg-beg end-end '(syntax-table . nil))
>                 ;; Mark the qualifier as attaching to the next sexp
>                 (put-text-property beg-beg beg-quote
>                                    'syntax-table
>                                    (string-to-syntax "'"))
>                 ;; Mark the quotes appropriately
>                 (put-text-property beg-quote beg-quote-after
>                                    'syntax-table
>                                    ;; (string-to-syntax "\"")
>                                    (string-to-syntax "|"))
>                 (put-text-property (1- end-end)
>                                    end-end
>                                    'syntax-table
>                                    (string-to-syntax "|"))))))))))

> ;; Then in a c++ buffer...
> (setq-local syntax-propertize-function #'c++11-syntax-propertize-function)

> -Ivan


> On Sun, Apr 3, 2016 at 12:36 PM, Alan Mackenzie <acm@muc.de> wrote:
> > Hello, Ivan.

> > On Tue, Mar 29, 2016 at 09:14:44PM -0600, Ivan Andrus wrote:
> >> Ivan Andrus <darthandrus@gmail.com> writes:

> >> > C++11 allows fancy new raw string literals [1], but these strings aren't
> >> > supported in c++-mode (e.g. fontification and movement by sexp's).

> >> > In my experience such raw strings are fairly rare, and they are no doubt
> >> > difficult to support.  But I thought I would report this since I
> >> > didn't see a bug for it in debbugs.

> >> > -Ivan

> >> > [1] http://en.wikipedia.org/wiki/C%2B%2B11#New_string_literals

> >> Any thoughts on this?  These are becoming more common in the code I work
> >> on and some colleagues and I would like support, since they can destroy
> >> fontification of the rest of the buffer.  I'm hesitant to try and do it
> >> myself because of the famed difficulty of cc-mode.  :-(  But I'm willing
> >> to try if someone has ideas.

> > OK, I'll have a go at adding these ASAP.  I've got a few ideas as how
> > best to do this.

> >> -Ivan

> > --
> > Alan Mackenzie (Nuremberg, Germany).