bug#13505: Bug#696026: emacs24: file corruption on saving

[Eli Zaretskii]

> Date: Mon, 21 Jan 2013 05:14:10 +0100
> From: Vincent Lefevre <vincent@vinc17.net>
> Cc: rlb@defaultvalue.org, handa@gnu.org, 13505@debbugs.gnu.org,
>       696026-forwarded@bugs.debian.org, 696026@bugs.debian.org
> 
> On 2013-01-21 05:48:14 +0200, Eli Zaretskii wrote:
> > > You said:
> > > 
> > > | The original encoded form of the characters as found on disk at
> > > | visit time _cannot_ be recovered by saving with raw-text, because
> > > | that encoded form is lost without a trace when the file is _visited_
> > >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > | and decoded into the internal representation.
> > > 
> > > This is what lossy is.
> > 
> > In that sense, every encoding except no-conversion is lossy.
> 
> Even 8-bit encodings such as latin-1?

Yes.  When latin-1 characters are decoded (as part of visiting a
file), they are converted to the internal representation, and cease to
be single 8-bit bytes.

> > > On the opposite, the utf-8 encoding doesn't seem to be lossy: Emacs
> > > seems to handle files with invalid UTF-8 sequences without any loss.
> > > So, this encoding is safe, even if Emacs wrongly guess the encoding.
> > 
> > No, it isn't, although you could get away with it most of the time.
> 
> Could you give an example where one loses data with the utf-8 encoding?

E.g., in your test file, the byte whose value is 0x80 is converted to
0x3fff80 when the file is read into a buffer.

Perhaps by "lossless" you mean "reversible", in the sense that saving
the same buffer will perform the reverse conversion.  In that case,
even the in-is13194-devanagari-unix is reversible: if you type this
encoding when Emacs prompts you to select one of the coding systems,
then you get the same file on disk with no corruption whatsoever.

> > > But Emacs should clearly tell the user what to do after C-x C-s and
> > > clearly say when there can be data loss.
> > 
> > At save time, "data loss" is wrt what's in the buffer.  In that sense,
> > the encodings Emacs suggested don't lose any data.
> 
> "data loss" is the difference between the original file and the saved
> file.

But what do you want Emacs to do with this?  When you save the buffer,
the original file might be different or no longer be available (or not
accessible even in principle, e.g. if the data came from a
subprocess).  These issues should be detected at file visit time, if
at all, not at buffer save time.

> > > Then Emacs says: "Select one of the safe coding systems listed below
> > > [...]", but doesn't say that something has already been lost. So, the
> > > words "safe coding systems" are really misleading.
> > 
> > It's misleading because you misunderstand what is "safe" at buffer
> > save time.
> 
> No, it's misleading because Emacs didn't say that data were lost
> when visiting the file.

Let's be constructive here.  Please suggest some practical way for
Emacs to handle this situation better.

For the record, here are the various alternative ways Emacs supports
the use case you described, when a file with inconsistent encoding
needs to be repaired manually:

 . Visit the file with "M-x find-file-literally RET".  This yields a
   unibyte buffer, where each byte stands for itself, and which you
   can edit without risking en-/decoding issues.

 . Visit the file normally, then type "M-x hexl-mode RET" (or use 
   "M-x hexl-find-file RET" to visit it in the first place).  This
   revisits (or visits) the file in a unibyte buffer, and in addition
   lets you edit the binary stuff regardless of its graphic
   representation.

 . After visiting the file normally and noticing that it contains
   weird characters, or after being prompted to select a coding system
   when saving the buffer, type "C-x RET r raw-text RET" to revisit
   the file in raw-text encoding.  Then edit the bytes and save the
   file.

These alternatives are listed in the descending order of priority
(IMO).  There are more ways to deal with this, but the rest are more
complicated and dangerous, so I don't mention them here.  (It is also
possible that you will find the second alternative more convenient
than the 1st one.)