I appreciate your hard work. I can imagine that it took you a lot of time, effort and agony to pull off this feature, and your efforts were worthwhile: you created a good and helpful feature. For some purposes the feature is perfect as it is, especially now that you've allowed customization of the paragraph separator. I believe Emacs can be the #1 plain-text bidi editor out there, but this hinges on fixing this bug.
> I maintain that Emacs deviates from the UBA in a relatively minor way,
in an aspect that is only tangentially related to reorderingbidirectional text for display, and that raises its head in situationsthat are relatively rare in practice, and in many of those rare casescan be easily worked around by breaking long lines.
One of the valuable aspects of an ISO standard is that it is not left to the personal judgment of a programmer to decide what is worth implementing, and how to do so. It is not for you to decide what is a minor detail and what is a major one, what is tangential and what is core. You need to implement it to the letter, or else you can't claim conformance, no matter how slight you imagine your deviation to be.
On what do you base your claim that this problem occurs relatively rarely in practice? This is the kind of statement that only a specialist linguist/statistician can make. And have you taken into account the type of demographics who use Emacs' bidi feature and the kinds of texts they're likely to type?
Contrary to what you said, my personal experience show that this is a major inconvenience, and that it is a situation that occurs very often, almost every paragraph, in fact, since I write primarily LaTeX documents where English markup is intermixed with predominantly Hebrew text containing frequent quotes from English textbooks and articles.
Yes, breaking lines is a possible workaround for LaTex, but it makes for ugly and erratic looking paragraphs that are difficult to read and edit. And what about documents that are not LaTeX? What workaround do they have?
You mention breaking "long lines", but this is not just a problem of long lines. It takes just two English words inside a Hebrew paragraph that happen to fall on a line break, to manifest this bug.
> even today, 10 years later, still shines among all the
bidi-aware editors out there, certainly among those of the FreeSoftware variety.
Yes, Emacs shines as one of the very rare bidi-aware text editors that enable entering explicit directional formatting characters. This is indeed to Emacs' credit and is a very helpful feature.
However, Emacs also shines as possibly the only bidi-aware text editor that botches the line wrapping of bidi paragraphs. Every single editor that I've checked gets it right: from Word to Kate to GEdit to Google Docs to BlueFish to TextEdit.
> And I don't really understand what is the purpose of your insistence
on the formal definition of this deviation. It certainly won't helpfixing this issue any time soon, not unless someone steps forward todo the job, which IMO is quite large.
I don't know what you mean by 'the formal definition of this deviation'. I think that Emacs should not mislead the users and potential users. That it should not claim to conform to a standard when it does not. I think that when prospective users google "Emacs bidi" or "Emacs unicode" they will be able to easily see that there's a problem with bidi line wrapping and that if they require a text editor that is Unicode compliant they should look elsewhere. The keywords are: transparency, truth in advertising, user-friendly, and standards-oriented.
> The Emacs manual already describes this deviation.
In the online manual sections 22.19 (Bidirectional Editing) and 37.26 (Bidirectional Display) claim that Emacs implements the Unicode Bidirectional Algorithm.