bug#9218: 24.0.50; bidi-display-reordering slowdown

[Eli Zaretskii]

> From: Florian Beck <abstraktion@t-online.de>
> Cc: Florian Beck <abstraktion@t-online.de>,  9218@debbugs.gnu.org
> Date: Mon, 01 Aug 2011 22:38:54 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Florian Beck <abstraktion@t-online.de>
> >> Cc: abstraktion@t-online.de,  9218@debbugs.gnu.org
> >> Date: Mon, 01 Aug 2011 21:34:30 +0200
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> >> Does setting bidi-paragraph-direction to left-to-right solve the
> >> >> problem?
> >> >
> >> > I meant without resetting bidi-display-reordering to nil, of course.
> >> 
> >> The file opens just as fast as with bidi turned of, yes. Scrolling is
> >> still somewhat erratic: scrolling down takes 12s the first time
> >
> > By "scrolling down" you mean all the way to the end, or just one
> > screenful?
> 
> `scroll-up-command'
> 
> And `end-of-buffer' takes the same time (12s vs 1s).
> 
> But it does not matter really. ANY command takes several seconds in that
> buffer, e.g. C-h k <up> or C-x C-b – whereas they are instantanious with
> bidi turned off.

And this slowdown happens only on the first command, as you said
earlier?  Or on subsequent commands as well?

> In fact <NEXT> may even be slowed down because of fontification going on
> (it only is slow the first time). Other commands are reproduciable.

"Reproducible" meaning that they are slow not only the first time?  Or
does it mean something else?

> Strange. C-h k is instantenious, but C-h k <up> takes several seconds to
> complete. Happens also when selecting the other window (*Help*) with the 
> mouse or
> when running C-h k <up> in the other window.

I'd expect the slowdown to happen whenever the window showing that Org
buffer needs to be redisplayed.  "C-h k" just displays the prompt in
the echo area, so the Org window needs not be redisplayed.  "C-h k
<up>" pops up another window, which changes the dimensions of the Org
window, and that window needs to be redisplayed.  Emacs tries very
hard to optimize the redisplay, because redisplaying everything would
be painfully slow.  That is the reason of what you see, IIUC.

> Once the large buffer is no longer displayed in any window, the speed
> returns to normal (as in before bidi).

Of course: a buffer not shown in any window does not affect redisplay
in any way.

> >> But that still requires me to set a per file variable.
> >
> > I think Org Mode should set bidi-paragraph-direction in all its
> > buffers.  Can you see any disadvantages to such a solution?
> 
> Personally, no. People writing right-to-left prose might, though.

Why would they?  setting bidi-paragraph-direction does not disable
reordering of R2L text, it just makes all Org entries start at the
left margin.

> I just don't understand why it has to be that slow. I have paragraphs of
> reasonable length. Is it because of the folded display?

Probably, but that's a guess, and I'm not very good at guessing when
the display engine is involved.

I don't suppose you can ship me that file for playing with it?  If
not, I will try to generate such a huge file and see if I get such
slowdown.  My Org files are much shorter (around 1.5K lines overall),
and I don't see any slowdown at all, even though I have Flyspell mode
turned on in them.

> (But then there are only some 6000 visible lines which should be
> fast in any case.)

The display engine has no way of knowing which parts are visible and
which aren't, until it actually traverses all of that text and takes
notice of the `invisible' text properties, selective-display, and
other display features.  This is much slower than just scanning buffer
text, and somehow the paragraph direction detection comes into play
during this in a way that slows things down unbearably...

> Org mode does quite a bit of fontification, i.e. it already parses
> the buffer.

Fontification pays attention only to a few text properties, while
redisplay needs to cater to much more.

> Is there a way for org mode to tell the display engine where a new
> paragraph should be considered, what is to be displayed RtL or LtR,
> etc?

How can it do that without running the same code that the bidi display
runs, and which is the cause of the slowdown?  The rules of
determining the paragraph direction are spelled out by an annex to the
Unicode Standard (http://unicode.org/reports/tr9/#The_Paragraph_Level),
and bidi.c includes its full implementation.  Any piece of code that
tries to do the same will be hit with the same slowness.

IOW, to solve this problem without the need to set
bidi-paragraph-direction non-nil, I need either (a) find a bug in the
current bidi display that makes paragraph detection so slow in this
case, or (b) find some way to optimize the paragraph detection so it
runs faster or is invoked less in most cases like this one.