Re: Compositions and bidi display

[Eli Zaretskii]

> From: Kenichi Handa <address@hidden>
> Date: Mon, 26 Apr 2010 20:29:18 +0900
> Cc: address@hidden
> 
> All composition-related functions expect characters are in
> logical order.

I assumed that much.  Sigh...

> So, the bidi reordering must happen after composition handling is
> done.

Unfortunately, this is impossible, not without throwing away the
entire design and current implementation of the bidi reordering, and
implementing it in a totally different way that will have to be much
more invasive into the overall design of Emacs display engine.

The reason is, as you know, that bidi reordering in Emacs is
conceptually just a replacement for advancing from one character to
the next during iteration through buffers or strings.  Instead of
incrementing the character position to the next character, we modify
the position non-linearly to get to the next character in the visual
order.  Obviously, this iteration is a lower-level operation than character
composition.

In addition, the bidi reordering engine knows nothing about the
characters it encounters except their bidirectional properties; in
particular, it doesn't know anything about character compositions, and
teaching it about them would mean rather serious complications.

Moreover, the bidirectional properties are in general defined for
individual characters, not for the composed ones, which is one more
reason it is very hard to do what you suggest, even if we would turn
the current design inside out.  For example, we compose Hebrew
consonants with diacriticals into a single glyph, but that glyph has
no character codepoint to look up its bidirectional properties in the
Unicode database.  So, once composed, these characters cannot be
reordered by following the UAX#9 algorithm without complications,
because UAX#9 is explicitly defined to work _before_ any shaping of
characters for display, see Section 3.5 there.

Therefore, I will need to find and handle sequences of characters to
be composed as an integral part of next_element_from_buffer, similarly
to what is already done with face changes there.

The idea is to detect the situation where the bidi iteration placed us
into a composable sequence of characters, and when that happens,
compose them and deliver them as a single display element, and then
skip the entire sequence, like we do today in the unidirectional
display.  The tricky part is that today we only detect this when we
hit the beginning of such a sequence, while moving in the strictly
increasing order of buffer positions; with bidi reordering we will
need to detect them from the end of the sequence as well, for when the
bidi iterator moves backwards or jumps across many character
positions.

Is it possible to write a function or macro that will find out, for a
particular buffer/string position, whether that position is at the end
or in the middle of a composable sequence of characters, and if so,
return the character positions of the first and last characters of the
sequence?  Something like CHAR_COMPOSED_P, but one that looks back in
the buffer?  If so, could you please help me write such a function?

TIA