Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <address@hidden>
> Cc: address@hidden
> Date: Thu, 25 Aug 2011 23:55:42 -0400
> 
> > It's possible.  It's called "embedding".  Here (watch the directional
> > control characters around the parentheses):
> >    Buffer name ‪(default "אבגדה"):‬ אבגדרשת123
> > Should we provide bidi-embed-string-l2r/r2l functions?
> 
> That sounds like just what we need, then.  How 'bout providing
> a (bidi-embed-string STR &optional DIRECTION) where DIRECTION works like
> bidi-paragraph-direction (i.e. nil means auto-detect).

Auto-detect is not really well-defined in this case, because the
DWIM-ish value of DIRECTION is determined by 3 factors:

 . the direction of the paragraph where STR will be inserted
 . the text around the position where it will be inserted
 . the desired effect of embedding the string

As you see, nothing here depends on the string itself.  The first 2
factors could be perhaps taken care of by providing additional
arguments that tell where the string will be inserted, although in
many cases the surrounding text will not be known yet (think inserting
a prompt into the minibuffer), at least not on both ends.  But the 3rd
factor is a complete mystery at this point, as I don't yet have enough
experience with using embeddings to enumerate all the possible use
cases.  If someone else can help produce such a list, I'm all ears.
Failing that, this function will have to be used by "experts" or after
some trial and error, and DIRECTION will have to be non-optional.

Alternatively, we could for now support nil as DIRECTION by assuming
the string will be inserted in the current buffer at point and using
the paragraph direction there.  It seems to DTRT in the simple use
cases I tried.  We could also add an optional BUFFER argument to use
the information from that buffer.

I guess what I'm saying is that the API will almost certainly change
as we gain experience.  So I'm unsure how elaborate we should make it
at this point, because any mistake will mean more changes in the
future when the API changes.

> >> Whether this ABC<1> appears within an L2R chunk of text or an L2R chunk
> >> of text.  I guess that "paragraph direction" is a good approximation.
> > Why not always display it is <1>CBA ?  That's the natural visual order.
> 
> To me, in an L2R context, any buffer name should be displayed as
> NAME<N>, regardless of whether NAME happens to be R2L or L2R (i.e. NAME
> should be an "embedded-string").

We will get what you want if we embed NAME in LRE..PDF when it's in an
L2R paragraph, and in RLE..PDF when it's in an R2L paragraph.  Or if
we insert LRM/RLM between NAME and <n>.  The difficulty is that you
need to know in advance the paragraph direction where the string will
be inserted.  This means the embedding should happen very close to
actually inserting the string.  IOW, utility functions such as
buffer-name cannot magically return a correctly decorated buffer name
and, what's worse, application code will need to take NAME<n> apart,
embed NAME or add the mark to it, then catenate <n> to it -- perhaps
we should have a function to do that.

There's a similar issue with file names: /abcd/ABCD/XYZ will be
displayed as /abcd/ZYX/DCBA.

Btw, using LRM/RLM is always preferred when possible, because it's
stateless, as opposed to an embedding.  UAX#9 says that much.  Also,
more single-byte encodings support the marks than the rest of the
directional controls.

So a related question is: should we prefer the marks where possible,
or should we standardize on embeddings?  I'm not sure we are equipped
with enough experience to decide at this time.  However, an important
practical question is: which alternative to use to fix read_minibuf,
for example?

Maybe we should embed fields and use the marks elsewhere?