Now: Paragraph Direction Detection and Harmonization -- Was: Re: Bidirectional editing in Emacs -- main design decisions

[Mohsen BANAN]

>>>>> On Mon, 25 Apr 2011 21:59:43 +0300, Eli Zaretskii <address@hidden> said:
  >> You see how the Farsi piece (word sequence -- not
  >> character sequence) is flipped around "Mohsen".

  Eli> Which one of the two is correct, in the original text or in the
  Eli> citations?  (I don't read Farsi.)

Of course, the original is correct.
Because I wrote it and I wrote it with emacs.

  Eli> I think the issue here is paragraph direction, which in Emacs is
  Eli> dynamically determined by default.  See below.

I agree. That is the problem.  

I have therefore changed the subject line to:
 Paragraph Direction Detection and Harmonization

Please see below as I expand on what parts I see
as non-problem and what parts I think can be
improved.

  >> In the browser my original text appears as it does
  >> in the citation.

  Eli> Does the browser display it flushed to the left or to the right?

Everything is flushed to the left. Which is
consistent your diagnosis of paragraph direction problem.

  Eli> Does it help to say "M-x set-variable RET bidi-paragraph-direction RET
  Eli> right-to-left RET" in Emacs?  Do you see both versions of the text
  Eli> correctly then?

Yes.

I have added key bindings for myself for
 (set-variable  'bidi-paragraph-direction 'right-to-left)
and 
 (set-variable  'bidi-paragraph-direction 'left-to-right)
for debugging. 

  >> And it displays just fine under emacs24 but word
  >> sequences are flipped when the browser renders
  >> that same byte sequence.

  Eli> So the Emacs display is correct, is that
  Eli> what you are saying?  

I am saying that emacs display is correct but that
there are interoperability problems.

  Eli> I don't care about Firefox, but I do care
  Eli> about the bidirectional display in Emacs.

The nature of bidi text is such that it demands
harmony.

For example, I think that it is worthwhile for
emacs24 to have a good Conformance Statement for 
 http://unicode.org/reports/tr9/

The existence of -- Unicode Standard Annex #9 --
Unicode Bidirectional Algorithm -- speaks to that
requirement for harmonization.

I looked in there for information about 
dynamic paragraph direction detection and did not
find much.

I think that the dynamic paragraph direction
detection in emacs needs improvement.

In the email citation the direction was guessed
wrong although that paragraph was dominantly RTL.

Can you please expand on what algorithm you use to 
determine dynamic paragraph direction?

With respect to:

  Eli> I don't care about Firefox

I have a different view. I see 
Emacs and Firefox as joint sisters.

See

   Blee: A Deeply Integrated Emacs User Environment For the Software-Service 
Continuum
   
http://linuxfestnorthwest.org/sessions/blee-deeply-integrated-emacs-user-environment-software-service-continum

for that perspective. 

A lot of benefits come from integrating the
browser with emacs.

With respect to Mozilla-Emacs display
inconsistency for bidi email generated with emacs,
I think the solution is to generate html and
specify paragraph direction explicitly in html.

Beyond the basic bidi capability in emacs, there
are several layers above it that we now need to
cultivate.

Thanks for your help.

...Mohsen
...محسن