Re: Long lines and bidi [Was: Re: bug#13623: ...]

[Eli Zaretskii]

> Date: Fri, 08 Feb 2013 20:21:57 +0400
> From: Dmitry Antipov <address@hidden>
> CC: address@hidden
> 
> On 02/08/2013 06:07 PM, Eli Zaretskii wrote:
> 
> > Profile alone is not enough.  Please tell how did you "scroll",
> > exactly (which commands did you use), and please also show the
> > absolute times it took to perform each command.
> 
> (defun scroll-both ()
>    (interactive)
>    (let ((start (float-time)))
>      (progn
>        (dotimes (n 100) (progn (scroll-up) (redisplay)))
>        (goto-char (point-max))
>        (dotimes (n 100) (progn (scroll-down) (redisplay)))
>        (message "Elapsed %f seconds" (- (float-time) start)))))
> 
> With bidi, ~600 second elapsed, and:
> 
>      25.18%        emacs  emacs                          [.] scan_buffer
>       7.04%        emacs  emacs                          [.] bidi_resolve_weak
>       6.47%        emacs  emacs                          [.] 
> get_next_display_element
>       6.37%        emacs  emacs                          [.] 
> bidi_level_of_next_char
>       5.14%        emacs  libc-2.16.so                   [.] 
> __memcpy_ssse3_back
>       5.05%        emacs  emacs                          [.] 
> move_it_in_display_line_to
>       4.94%        emacs  emacs                          [.] x_produce_glyphs
>       4.84%        emacs  libXft.so.2.3.1                [.] XftCharIndex
>       3.72%        emacs  emacs                          [.] 
> bidi_move_to_visually_next
>       3.70%        emacs  emacs                          [.] 
> next_element_from_buffer
>       2.90%        emacs  libXft.so.2.3.1                [.] XftGlyphExtents
>       2.05%        emacs  emacs                          [.] bidi_fetch_char
>       2.02%        emacs  emacs                          [.] 
> lookup_glyphless_char_display
>       2.01%        emacs  emacs                          [.] 
> bidi_resolve_neutral
>       1.76%        emacs  emacs                          [.] 
> bidi_cache_iterator_state
>       1.70%        emacs  emacs                          [.] bidi_get_type
>       1.51%        emacs  emacs                          [.] 
> bidi_resolve_explicit_1
>       1.18%        emacs  libXft.so.2.3.1                [.] XftFontCheckGlyph
>       1.12%        emacs  emacs                          [.] 
> xftfont_encode_char
>       1.01%        emacs  emacs                          [.] 
> xftfont_text_extents
> 
> Without bidi, ~230 seconds elapsed, and:

This is consistent with my past measurements:

 (a) disabling bidi makes redisplay faster, but it is still awfully
     slow (2.3 sec per scroll);

 (b) bidi iteration is about 2 times slower than the unidirectional
     one (you get 3 times slower because your buffer is full of weak
     characters, which make the bidi iterator work harder due to the
     requirements of the Unicode Bidirectional Algorithm.

> I suspect that scroll should be direction-agnostic in theory

That theory is wrong.  The reason is that functions that move by
display lines can only move forward.  So moving backward is coded very
differently (a.k.a. "slower").

> but both profiled runs shows that scroll-down is much, much slower
> than scroll-up (that's why elapsed time is so huge in both cases).

That's expected; see also my explanation in a previous mail, which
describes what move_it_vertically_backward does.  That function is
used a lot by scroll-down.

> > What was in the file?  bidi_resolve_weak high on the profile hints
> > that it was full of punctuation or digits or banks, which is not
> > really an interesting case.
> 
> Your guess is correct; but I suspect that an average text in human language
> contains less punctuations, digits and blanks than the C source code of the
> same size :-).

An average C code still has only a small fraction of punctuation.
Just look at any C file.

> > As to your question: how can we know what characters are or aren't in
> > the buffer without scanning it?  And scanning the buffer is exactly
> > what bidi.c does.
> 
> Hm... insert-file-contents tries to detect encoding by looking at first 1K
> and last 3K of the file. Why the similar approach isn't applicable to bidi?

No.  Detecting encoding by a small portion is a heuristic that works
only because most every file is encoded consistently.  When a file is
encoded inconsistently, the result of the above decoding heuristic is
horribly wrong, and the consequences for the user are grave.  As a
recent example, see bug #13505.

By contrast, scripts used in a text file do not have to be consistent
or uniformly distributed over the file at all.  So the probability to
get this wrong will be much higher.