bug#33092: 26.1; Re-running shell wipes font-lock from xterm-color

[Allen Li]

Where to begin?

The problem I am having is that when there is already a *shell* buffer
without a running process, re-running the `shell' command will reuse the
buffer and wipe the font-lock color added by xterm-color.el, a
third-party package.

I'm reporting this here because it seems like Emacs could be improved in
some way to avoid this class of problem.

What causes this is as follows:

The `shell' command will call `make-comint-in-buffer' and `shell-mode'.

`shell-mode' will then call `comint-mode' as a derived mode.

`comint-mode' will call `kill-all-local-variables'.

`comint-mode' also adds `font-lock-defontify' to
`change-major-mode-hook'.  Since we're already in `shell-mode' derived
from `comint-mode', switching "back" to `shell-mode' wipes font-lock.

I can work around this pretty easily in my personal config, by removing
`font-lock-defontify' from `change-major-mode-hook' with a well-placed
advice, but the current behavior doesn't seem comfortable to me.

First, if we're reusing the buffer, do we have to switch to `shell-mode'
again?  The advantage is that it clears out some buffer state, so it's
kind of like running a "fresh" `shell'.

But the existing output in the buffer isn't cleared (although it is
defontified as I found out to my dismay).  The `default-directory' isn't
reset either; normally running `shell' starts the shell with the
`default-directory' of the buffer of your current window, but not if it
reuses a *shell* buffer.

So the current behavior of `shell', if it reuses the *shell* buffer, is
to do a half-hearted reset of the buffer state.  As a user this is very
hard to understand and I certainly would not have understood it had I
not taken Edebug to the problem.

`shell' should either fully reset the buffer or touch as little as
possible.  The reset option is indistinguishable from killing the existing
*shell* buffer and creating a new one.  Therefore, if we are going to
reuse the buffer, I think the desired intent would be to touch as little
as possible, in which case we want to avoid the `shell-mode'
reset if we are reusing the *shell* buffer.

In GNU Emacs 26.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.22.30)
 of 2018-07-05 built on juergen
Windowing system distributor 'The X.Org Foundation', version 11.0.12001000

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft --with-modules
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong
 -fno-plt' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY
ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS
GTK3 X11 MODULES THREADS LIBSYSTEMD LCMS2

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix