bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Phil Sainty]

This is a spin-off from bug#20611: 24.4; mutt slow in ansi-term.

I'm logging a new bug report by request, although it seems to me that
this is still the same bug (and 20611 is still open), so I suggest that
they can both be closed together once a fix is committed?

I've experimented further with this issue since my comments in 20611,
and my current observations follow...

In certain circumstances, a full-screen redraw in a term.el terminal is
incredibly slow, apparently increasing with the (width x height) number
of characters in the terminal -- certainly things are dramatically worse
when the term window is full-screen, than when it is only small; and as
observed below, the slow-down is worse than linear with respect to the
number of characters being drawn -- each block of characters drawn takes
noticeably more time to draw than the previous block did).

The issue affects both GUI and terminal Emacs, and affects all terminal
buffers generated by term.el (i.e. this is not specific to `ansi-term'
as it was treated in the original bug, but will also affect `term' and
any other similar wrapper command which creates a terminal and runs a
given program in it.)

In my case a full-screen terminal is (210x48) characters, for which the
initial screen redraw when running Debian package manager command
"dpkg-reconfigure postfix" takes ~7 seconds (with the redraw occurring
in three 'blocks' of visible activity, each of which 'instantly' draws
the next X characters of the display, followed by some seconds of
waiting before the next burst of visible activity.

Conversely with a terminal size of (80x21) characters, performance is
much better, with a redraw taking ~1 second and not obviously divided
into blocks, but all happening at once. I would say that the block
size seen in the large terminal would exceed the number of characters
visible in the small terminal, so it is not surprising to see it all
drawn in a single block of activity.)

Curiously, I've just noticed that the redraw in the small case takes
~1 second usually; but if I expand the window and cause a slow redraw,
and then reduce it to the original size and try again, the redraw is
now much slower (~2 seconds instead of ~1).

My speculation is that the amount of preceding text in the buffer is
having an effect, because switching from a large window to a small
window meant that a lot of the text which had been visible in the large
window was no longer visible, yet still present if you scrolled back in
the buffer.

I've seemingly confirmed this by repeating the process (1. small and
quick-ish; 2. large and extremely slow; 3. small again and not as quick
as (1)); then deleting the contents of the buffer prior to the final
shell prompt and repeating the command a fourth time, for which I once
again observed a quick-ish response as per (1).

Likewise, adding a lot of preceding text into a large-sized term buffer
caused the command to slow down even more. Testing in GUI Emacs with an
even larger area (234x53) I was seeing ~10 second redraws initially, and
that increased to ~17 seconds after dumping a bunch of other text into
the buffer.

Furthermore, I see now that the processing time increases for each
block displayed during the redraw. I assume all but the final block has
the same number of characters, yet the approximate elapsed time when
each appears on screen is as follows:

~1 second
~4 seconds
~9 seconds
~10 seconds (n.b. the final block is much smaller than the others)

I believe those first three blocks are the same number of characters,
yet the processing times needed to draw them are roughly 1, 3, and 5
seconds respectively.

(At this point I would hazard a guess that each character drawn is also
processing, to some extent, all of the preceding characters?)

I haven't tried to delve into the term.el code at all (and I am not
familiar with terminal emulation in any case), so I can only hope that
these observations might help to explain the root cause of this issue.

As established in bug#20611, disabling the bidi support (by either
setting bidi-paragraph-direction to 'left-to-right, or by setting
bidi-display-reordering to nil) improves the speed of the slow redraws
by an order of magnitude.

The proposed fix for that bug sets bidi-paragraph-direction in the
`ansi-term' command, but this is not general enough. My initial thought
is to set the value in `term-mode' instead.

I am not familiar with bidi concerns, and do not know whether there is
ever a use-case for bidi being active in a term-mode buffer, but I note
the following comment in the change committed for the original bug:

  ;; There's a larger problem here with supporting bidirectional text:
  ;; the application that writes to the terminal could have its own
  ;; ideas about displaying bidirectional text, and might not want us
  ;; reordering the text or deciding on base paragraph direction.  One
  ;; such application is Emacs in TTY mode...  FIXME.



-Phil









In GNU Emacs 25.0.95.1 (x86_64-unknown-linux-gnu, X toolkit, Xaw3d 
scroll bars)
 of 2016-06-18 built on shodan
Repository revision: 2ad3d0182df68f00cca57584807f721c3c5c96f1
Windowing system distributor 'The X.Org Foundation', version 11.0.11701000
System Description:     Ubuntu 15.04

Configured using:
 'configure --prefix=/home/phil/emacs/trunk/usr/local
 --with-x-toolkit=lucid --without-sound'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK DBUS GSETTINGS NOTIFY
GNUTLS LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11

Important settings:
  value of $LANG: en_NZ.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
GNU Emacs 25.0.95.1 (x86_64-unknown-linux-gnu, X toolkit, Xaw3d scroll 
bars) of 2016-06-18
Mark activated

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util help-fns help-mode easymenu
cl-loaddefs pcase cl-lib mail-prsvr mail-utils time-date mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote dbusbind inotify dynamic-setting
system-font-setting font-render-setting x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 88249 8416)
 (symbols 48 20114 0)
 (miscs 40 46 146)
 (strings 32 15667 3956)
 (string-bytes 1 429343)
 (vectors 16 11737)
 (vector-slots 8 429691 5060)
 (floats 8 166 283)
 (intervals 56 223 0)
 (buffers 976 11)
 (heap 1024 29329 1004))