I added a couple of related fixes, put a snapshot in
ftp://invisible-island.net/temp/vile-9.6j2.tgz
OK, that behaves as you say it should (and basically as I expect) by
displaying \?A3 for the invalid pound signs.
It also highlights a problem with the XFCE Terminal which can't change
character set under user control. The gnome-terminal can change at
user request and allowed me to test this easily by creating a file
with iso-8859-1 pound signs in it and then switching back to utf-8 to
see how it behaved. So it's gnome-terminal for me from now on! :-)
I'll keep running the 9.6j2 code and report any other glitches I come
across.