bug#22320: Overlays with an 'invisible property break stacking of overlay faces

[Clément Pit--Claudel]

Hi Eli,

Thanks a lot for your response.

On 01/07/2016 10:54 AM, Eli Zaretskii wrote:
> It's a feature.  More accurately, it's the best solution found (more
> than 10 years ago!) for a difficult problem.  In a nutshell, in a
> general use case the invisible text can (and many times does) have
> non-trivial face attributes, and using those faces for displaying the
> ellipsis would result in each instance of the ellipsis to have some
> random face.  This "feature" makes that face predictable: when the
> invisible text has a face that is different from the last visible
> character before it, the ellipsis is displayed with the 'default'
> face.
>
> You can read the discussion which led to the current implementation
> here:
> 
>   http://lists.gnu.org/archive/html/emacs-pretest-bug/2005-04/msg00338.html

Interesting, thanks for the pointer. I read the discussion, but I see no 
rationale there for this choice. Was one ever given? Why is this better than, 
for example, keeping the face of the first invisible character and extending it 
to the whole ellipsis?

Am I mistaken to think that this choice is also inconsistent with the way faces 
on composed characters are handled? It seems that composing a sequence of 
characters into "..." will keep the face of the first compose character. That 
default is useful in many places (such as Arthur's nameless mode, or in 
flyspell or flycheck when an error covers a sequence of characters composed 
into a single one). Is there a good reason to treat ellipses standing for 
invisible spans differently?

The interactions of the current behaviour with transient mark mode are 
especially confusing: consider the two examples below:

(with-current-buffer (get-buffer-create "region covers "def", but ellipsis 
isn't highlighted")
  (erase-buffer)
  (fundamental-mode)
  (add-to-invisibility-spec '(outline . t))
  (insert "abcdefghi")
  (let ((ov (make-overlay 4 7)))
    (overlay-put ov 'invisible 'outline))
  (let ((ov (make-overlay 6 (point-max))))
    (overlay-put ov 'face 'region))
  (pop-to-buffer (current-buffer)))

(with-current-buffer (get-buffer-create "region now covers one more char, and 
ellispis is highlighed")
  (erase-buffer)
  (fundamental-mode)
  (add-to-invisibility-spec '(outline . t))
  (insert "abcdefghi")
  (let ((ov (make-overlay 4 7)))
    (overlay-put ov 'invisible 'outline))
  (let ((ov (make-overlay 5 (point-max))))
    (overlay-put ov 'face 'region))
  (pop-to-buffer (current-buffer)))

Using transient-mark-mode and selecting characters one by one from the end of 
the buffer, "i", "h", and "g" are progressively selected, then nothing happens 
despite "def" being in fact selected (so pressing C-w at that point kills more 
than highlighted), then "def" and "c" get highlighted at the same time. This 
means that the highlighted region doesn't actually correspond to the marked 
region. Is that also expected? (I realize that this can also happen if the 
point is in the middle of an invisible region, but that's a lot more of a 
corner case than using Shift+arrow keys to select a buffer section)

> Note that the display engine only looks at the face of the first
> invisible character; the rest are skipped without looking at their
> faces (or any other text properties or overlays).  So if you change
> your test case in a very minor way:
> 
>   (let ((ov (make-overlay 8 9)))
>     (overlay-put ov 'face '(:underline t)))
> 
> The "problem" doesn't happen.

That's true, but I have no control over this; the only thing that I see is that 
invisible sections of org Buffers do not have the right background (same for 
folded theorems in Coq buffers, due to the bug with font fallback and overlays).

>> Here is another (more "real-world") way to reproduce this issue (run the
>> snippet, then use C-x h in the newly created buffer to mark all text: the
>> invisible section isn't highlighted)
>>
>> (with-current-buffer (get-buffer-create "org-bug")
>>   (erase-buffer)
>>   (org-mode)
>>   (insert "* line1\n** line2\n* line3")
>>   (org-shifttab)
>>   (pop-to-buffer (current-buffer)))
> 
> You say that this use case is only broken in Emacs 25, but I see this
> in all the previous versions as well (as expected, given when this was
> coded).  So if you really see this working correctly in Emacs 24.4,
> then some other factors on your system were at work here.  Maybe you
> didn't test this exact test case there?

You're right; this example works fine in 24.3, but it is indeed broken in 24.5. 
I tested both with -Q:

   Works:  GNU Emacs 24.3.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8) of 
2015-06-17 on clem-w50-mint
   Broken: GNU Emacs 24.5.2 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8) of 
2015-06-20 on clem-w50-mint

Is this something that changed in org then?

>> Yet another way it to reproduce it is given below
>>
>> (with-current-buffer (get-buffer-create "flyspell-bug")
>>   (erase-buffer)
>>   (text-mode)
>>   (add-to-invisibility-spec '(outline . t))
>>   (insert "line1\nline2\nline3")
>>   (flyspell-check-region-doublons (point-min) (point-max))
>>   (let ((ov (make-overlay 7 12)))
>>     (overlay-put ov 'invisible 'outline))
>>   (pop-to-buffer (current-buffer)))
> 
> What exactly is the problem here?  I don't see anything that looks
> problematic.  In particular, there are no duplicate misspelling in
> this buffer, so flyspell-check-region-doublons should do nothing.
> What am I missing?

Indeed, I use aspell instead of ispell. It seems to ignore numbers, and thus 
flags the first two instances of "line" as duplicates.

>> Interestingly, the first example is broken in both 24.4 and emacs-25; but 
>> the 
>> second one is broken only in emacs-25 (the third one does not run in Emacs 
>> 24.4).
> 
> This behavior should be present in any Emacs released after Apr 2005.

I wonder what makes 24.3 behave differently in Org mode.

>> I wonder if this is connected to another issue that I've seen, where having a
>> composition (using prettify-symbols-mode) just before an invisible region 
>> would 
>> cause the background of the invisible region to be incorrect, but setting 
>> prettify-symbols-unprettify-at-point and moving the point to the edge of the 
>> symbol would cause the background to be fine. Unfortunately I don't have a 
>> good 
>> repro for that issue.
> 
> Hard to tell without a reproducible test case.

This is in fact due to font fallback. I opened a separate bug report, as it 
does not seem obviously related to the current issue.

Thanks,
Clément.

signature.asc
Description: OpenPGP digital signature