Handling invisible text in bidirectional display

[Eli Zaretskii]

This is another design decision I needed to make -- how to handle
invisible text in the display engine modified to support bidirectional
text.

The current unidirectional display code simply skips all the
characters that have the invisible property (or are covered by an
overlay with such a property).  The iteration then resumes at the
first character that does not have the invisible property.  Thus,
Emacs behaves as if the invisible characters simply did not exist in
the buffer.

But with bidirectional display, the invisible property can begin
and/or end in the middle of text that is reordered for display.  For
example, suppose the buffer includes the following text (capital
letters stand for right-to-left characters):

  abcABCxyz

This would normally be displayed like this:

  abcCBAxyz

Now imagine that we have an invisible property on characters C and x.
The current code would skip C and x, and stop on y.  But because C is
encountered when moving backwards from C to A, that causes us to miss
A and B entirely and display

  abcyz

which is clearly incorrect.  There are additional complications when
embedding characters such as RLE and LRO are involved, to override the
normal visual order of characters -- in these cases, a suitably placed
invisible property can change the visual order of the surrounding
visible characters, sometimes in a prominent way.

Therefore, I decided to modify the code which skips invisible
characters, such that it will skip them in the visual order, until it
finds the first character that is outside the region covered by the
invisible property.  Note that the ``first'' visible character could
be before or after the invisible region, in the logical order.

This preserves the visual order of displayed characters and does not
miss any characters that should be visible, but it does have
consequences that are not necessarily trivial.

One consequence is that a run of invisible characters can now be split
into several non-contiguous runs.  For example, this text:

  abcABCxyz

with c and A covered by an invisible property will be displayed as

  abCBxyz

where invisible text is split in two: one after b, the other before x.

If ellipsis should be displayed instead of the invisible text, we will
now have two places with ellipsis: one instead of c, the other instead
of A.

To me, these effects seem reasonable.  MS Word behaves the same, but
it doesn't have the ellipsis feature.  And since I'm not aware of any
other application except MS Word that supports both bidirectional text
and invisible text, I don't know what other reasonable behaviors can
be programmed.  If someone knows, please tell.

Eventually, all this boils down to the question of the use-cases that
involve invisible text.  If someone can post a list of such use-cases,
I could think whether the above modification of the display code will
support them.

TIA