Re: Unicode support

[Eli Zaretskii]

> Date: Tue, 24 Jul 2001 17:44:47 +0200 (CEST)
> From: Werner LEMBERG <address@hidden>
> 
> > We discussed similar issues related to bidirectional support with
> > Gerd.  The current state of thinking, as I understand it, is that
> > the rearrangement of characters for display (i.e. conversion between
> > the logical and visual order) should happen inside the functions
> > which walk the buffer and supply the next character for display to
> > the next layer (which generates glyphs).
> 
> Eli, you are the expert here.

Only compared to some ;-)

I wish I had more knowledgeable people to consult with, who would know
both about Emacs redisplay engine and bidi-specific issues.  Everybody
who is involved, including Gerd, Handa-san, and the people who
subscribe to the emacs-bidi mailing (including Roozbeh, btw ;-) are
extremely helpful, but I'm left alone when I need to bring all their
separate branches of knowledge together.  Which makes me petrified to
think that I might make some grave design mistake which will later
haunt Emacs...

> Maybe it is possible to use some freely available Unicode BIDI
> algorithms to avoid the reinvention of the wheel.

This was considered, but it is not easy.  All the implementations of
the Unicode bidi algorithm known to me process characters en masse, in
more-or-less large blocks.  That is, you hand them a buffer of text in
logical order, and get back that text reordered for display (``visual
order'').  By contrast, the current design principle, as suggested by
Gerd based on his intuition about the display engine, calls for an
algorithm that could be called sequentially, once for each character
that is being converted into a glyph.  This requires to turn a typical
implementation of UTR#9 ``inside out''.

Maybe this is not clear enough, so let me explain it from a different
perspective.

At the heart of the current Emacs 21 display engine there are 2
functions: get_next_display_element and set_iterator_to_next.  The
former takes whatever is at the current position in the buffer (let's
assume it's a character, but it can be an image or something else) and
produces from it a structure, called `struct glyph', that contains all
the information needed to display that character.  The other function,
set_iterator_to_next, moves to the next character to be displayed.
When Emacs needs to display a part of text, it marches through that
text with this pair of functions, producing the glyph structures one
by one as it goes.  (To complete the picture: those glyph structures
are stored in ``glyph matrices'' which are compared to the glyph
matrices that hold the snapshot of the current display, and the
differences between those two matrices are used to decide which parts
of the screen need to be redrawn.)

Right now, set_iterator_to_next simply advances through the buffer to
the next character in the logical order (it also handles overlays,
text properties, and other complications, but let's forget about that
for now).  What I'm trying to do is to make set_iterator_to_next
advance to the character that should be _displayed_ next, i.e. to find
the next character in the _visual_ order.  Clearly, this requires that
the implementation of UTR#9 be callable for each character in turn,
instead of processing many characters in one go.  This is unlike every
UTR#9 implementation I've seen.

Do I make any sense?