bug#3824: 23.1.50; too much effort is put into handling Scheme S-expression comments, causing problems

[Taylor R Campbell]

(Apologies for duplicates: apparently sending mail from this machine
using Emacs is non-trivial.  Grmble.)

Consider the following line of Scheme code:

   (foo bar #;(baz (quux #;() zot) mumble) frotz)

If the point is at the beginning, hitting C-M-f causes Emacs to barf
on imbalanced parentheses.  This is because Emacs goes to excessive
effort to handle S-expression comments in Scheme Mode, which causes
more problems than it solves.

Emacs should treat `#;' as whitespace, nothing more.  The text
following `#;' must be a valid S-expression anyway, so treating it as
if it were a comment, which can contain any unstructured text except
for a comment ender, leads to trouble.  For example, because Emacs's
`parse-partial-sexp' says that the point is in a comment if it is in
an S-expression comment, paredit fails to preserve structure there.
This is an extreme example; Emacs's S-expression motion commands in
general should work inside S-expression comments, but they don't.  For
example, if `|' denotes the point in

   (foo bar #;(baz |(quux #;() zot) mumble) frotz),

then C-M-f should cause the point to turn up at

   (foo bar #;(baz (quux #;() zot)| mumble) frotz),

but instead it barfs on imbalanced parentheses.

As far as I can imagine, the only legitimate context for treating `#;'
as more than whitespace is Font Lock, but since Font Lock works with
regular expressions, it, too, will do the wrong thing.  (In the above
example, Emacs fontifies everything after the initial `#;' as a
comment, including the text `frotz)', which is outside the comment.)

If it is too hard to make Emacs treat `#;' as whitespace and nothing
more, then Emacs shouldn't treat the octothorpe specially, and simply
read the rest of the line as a line comment.  Scheme programmers can
then just break the line after the semicolon.  Emacs's current attempt
to be clever causes trouble even for this simple workaround.  For
example, if `|' denotes the point in

   (foo bar
        #;
       |(baz (quux #;
                   () zot) mumble) frotz),

then C-M-f causes the point to move to the very end of the whole text,
rather than to

   (foo bar
        #;
        (baz (quux #;
                   () zot) mumble)| frotz),

where it should go.

In GNU Emacs 23.1.50.1 (i386-apple-darwin9.7.0, GTK+ Version 2.12.9)
 of 2009-07-11 on Oberon.local
Windowing system distributor `The X.Org Foundation', version 11.0.10402000
Important settings:
  value of $LC_ALL: en_US.UTF-8
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: nil
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Scheme

Minor modes in effect:
  show-paren-mode: t
  tooltip-mode: t
  tool-bar-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
S-SPC f o o <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> # | f o o <backspace> <backspace> 
<backspace> SPC f o o SPC | # C-b C-b C-b C-b M-: M-p 
<return> <help-echo> <help-echo> C-b C-b C-b C-b C-b 
C-b C-b C-b C-b C-b C-b C-b C-b C-b b <backspace> C-b 
C-b SPC ( f r o b b l e ) S-SPC C-M-u C-M-f C-b C-b 
C-b C-b C-b C-b C-b C-b C-b C-b C-b C-b C-b C-b C-b 
C-b C-b C-b C-b C-b C-b C-b C-b C-b C-b C-b C-b C-b 
C-b C-b C-b C-b C-b C-M-f C-M-b C-M-f C-M-b C-M-f C-M-b 
C-M-f C-b C-b C-b C-b C-b C-b C-b C-b C-b C-b C-b C-b 
[ ] C-a C-M-f C-M-b C-M-f C-M-b C-M-f C-b C-b C-b C-b 
C-b C-b C-b C-b C-b C-b C-b C-b C-b C-b C-b C-f SPC 
# ; C-f C-f SPC C-a C-M-f C-M-b C-M-f C-M-f C-M-f C-M-f 
C-M-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f 
C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-d 
C-d ( ) C-a C-M-f C-M-f C-M-f C-M-b C-M-f C-M-b C-M-f 
C-M-b C-M-f C-M-b C-M-f C-M-b C-M-f C-M-b C-M-f C-M-b 
C-M-f C-M-b C-M-f C-M-b C-M-f C-M-b <help-echo> C-a 
C-n C-p C-n C-p C-f C-e C-b C-b C-b C-b C-b C-b C-b 
C-b C-b C-b C-b C-b C-k C-n C-a C-p M-f M-f M-f z M-f 
<M-backspace> q u u x M-f <M-backspace> z o t M-f <M-backspace> 
m u m b l e M-f <M-backspace> f r o t z C-n C-x o C-x 
4 0 M-x r e p r o t SPC e m a <backspace> <backspace> 
<backspace> <backspace> <backspace> <backspace> o r 
t SPC e m a c s SPC b u g <help-echo> <help-echo> <help-echo> 
<return>

Recent messages:
(1 1 18 nil nil nil 0 nil nil (1))
(1 1 6 nil nil nil 0 nil nil (1))
(1 1 6 nil t nil 0 nil 10 (1))
(1 1 6 nil nil nil 0 nil nil (1))
let: End of file during parsing
(1 1 6 nil t nil 0 nil 10 (1))
Undo!
(0 nil 1 nil t nil 0 nil 26 nil)
(0 nil 1 nil 1 nil 0 t 26 nil)
forward-sexp: Scan error: "Unbalanced parentheses", 1, 53 [18 times]