bug#8218: 23.2; scan-sexps sees inconsistent syntax

[Stefan Monnier]

> I am seeing (scan-sexps) show inconsistent behavior depending on WHEN it
> is evaluated. If the attached sample file is opened, the point placed at
> the first '{', and (scan-sexps (point) 1) evaluated, the returned
> position is at times of the first '}' (wrong) and at other times of the
> second '}' (right). If the evaluation is done right when the file is
> opened, I observe the wrong behavior. This is reliably triggered with

> $ emacs -Q /tmp/tst3.pm --eval '(goto-char (scan-sexps 5 1))'

> Any subsequent evaluation of the same form in that buffer produces the
> correct behavior. If I turn off font-lock-mode, I consistently get
> incorrect behavior. This has wide ramifications with occasional
> incorrect indentation and hide-show operation, for instance.

This is not a bug in syntax.c: the syntax of Perl is too complex for the
simple syntax-table mechanism, so it needs to use the escape hatch,
i.e. the `syntax-table' property.  This property is added to particular
parts of the text via font-lock-syntactic-keywords (in Emacs-23) and via
syntax-propertize (in Emacs-24).  So in Emacs-23, you will only get the
right answer if font-lock is enabled and has processed the relevant
parts of the buffer.  In Emacs-24, it can work right even if font-lock
is disabled, but you still need to make sure the relevant parts of the
buffer have been processed.  For that, you should call
(syntax-propertize POS) where POS is the upper bound of the chunk of
text you're interested.


        Stefan