bug#18355: unicode and multi-byte key-event support for function recent-keys

[Boruch Baum]

SUMMARY: I found a bug downstream in Debian's emacs-goodies-el package
(file home-end.el) that, in turn, reveals a deficiency in the way
function `recent-keys' represents keyboard-events. The documentation for
recent-keys claims that it returns information for 300 keyboard
events. However, `recent-keys' returns a single 300 element (byte)
vector, into which are pushed multi-byte events. This is a problem even
without devolving into the morass that is unicode. For example, my USA
English keyboard generates a three byte event for [home] and [end]
key-presses.

DESIRED STATE: Function `recent-keys' should return a list of
variable-length vectors, one vector per keyboard event.

DISCUSSION: Function `recent-keys' returns a 300-byte vector of most
recent keyboard events, and describes this as the 300 most recent
keypresses. However, between the advent of unicode variable-byte locales
and modern keyboards, those 300 bytes represent an unknown number of
keyboard events, that become very difficult to parse. There is no
guarantee that the bottom of the FIFO corresponds to the first byte of a
keyboard event, and there isn't clarity how many bytes at the top
of the FIFO are the first event, since it is the FINAL byte of the most
recent event that is on the top of the FIFO..

HOW I CAME ACROSS THIS: Debian's emacs-goodies-el package includes a
file home-end.el, whose components didn't work for me. The functions
were assuming that the three top-most bytes of the recent-keys vector
were three separate key events, and that the [home] and [end] keys were
generating single-byte events. However, my English USA keyboard
generates bytes '27 79 72' (ESC-O-H) for [home], and bytes '27 79 70'
(ESC-O-F) for [end].

I have a downstream patch prepared for the specific case of home-end.el,
and plan to prepare today debian and developer bug reports.

In GNU Emacs 24.3.1 (x86_64-pc-linux-gnu)
 of 2014-06-06 on barber, modified by Debian
System Description:     Debian 7.0 GNU/Linux wheezy/testing

Configured using:
 `configure '--build' 'x86_64-linux-gnu' '--build' 'x86_64-linux-gnu'
 '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib'
 '--localstatedir=/var/lib' '--infodir=/usr/share/info'
 '--mandir=/usr/share/man' '--with-pop=yes'
 
'--enable-locallisppath=/etc/emacs24:/etc/emacs:/usr/local/share/emacs/24.3/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/24.3/site-lis\
p:/usr/share/emacs/site-lisp'
 '--with-crt-dir=/usr/lib/x86_64-linux-gnu' '--with-x=no'
 '--without-gconf' '--without-gsettings' 'build_alias=x86_64-linux-gnu'
 'CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat
 -Werror=format-security -Wall' 'LDFLAGS=-Wl,-z,relro'
 'CPPFLAGS=-D_FORTIFY_SOURCE=2''

Important settings:
  value of $LANG: en_US.utf8
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Help

Minor modes in effect:
  desktop-save-mode: t
  savehist-mode: t
  winner-mode: t
  show-paren-mode: t
  global-hi-lock-mode: t
  hi-lock-mode: t
  delete-selection-mode: t
  shell-dirtrack-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Auto-saving...done
Auto-saving...done
Auto-saving...done
Saving file /home/optimum/emacs-bug-report...
Wrote /home/optimum/emacs-bug-report
Auto-saving...done
line-move-visual: Beginning of buffer [7 times]
Auto-saving...done
Type "q" in help window to restore its previous buffer.
mouse-2, RET: find function's definition
Quit

Load-path shadows:
~/.emacs.d/elpa/w3m-20140107.1708/w3m-favicon hides
/usr/share/emacs24/site-lisp/w3m/w3m-favicon
~/.emacs.d/elpa/w3m-20140107.1708/w3m-symbol hides
/usr/share/emacs24/site-lisp/w3m/w3m-symbol
~/.emacs.d/elpa/w3m-20140107.1708/bookmark-w3m hides
/usr/share/emacs24/site-lisp/w3m/bookmark-w3m
~/.emacs.d/elpa/w3m-20140107.1708/mime-w3m hides
/usr/share/emacs24/site-lisp/w3m/mime-w3m
~/.emacs.d/elpa/w3m-20140107.1708/w3m-filter hides
/usr/share/emacs24/site-lisp/w3m/w3m-filter
~/.emacs.d/elpa/w3m-20140107.1708/w3m-ccl hides
/usr/share/emacs24/site-lisp/w3m/w3m-ccl
~/.emacs.d/elpa/w3m-20140107.1708/w3m-namazu hides
/usr/share/emacs24/site-lisp/w3m/w3m-namazu
~/.emacs.d/elpa/w3m-20140107.1708/w3m-ems hides
/usr/share/emacs24/site-lisp/w3m/w3m-ems
~/.emacs.d/elpa/w3m-20140107.1708/octet hides
/usr/share/emacs24/site-lisp/w3m/octet
~/.emacs.d/elpa/w3m-20140107.1708/w3m-bookmark hides
/usr/share/emacs24/site-lisp/w3m/w3m-bookmark
~/.emacs.d/elpa/w3m-20140107.1708/w3m-perldoc hides
/usr/share/emacs24/site-lisp/w3m/w3m-perldoc
~/.emacs.d/elpa/w3m-20140107.1708/w3m-dtree hides
/usr/share/emacs24/site-lisp/w3m/w3m-dtree
~/.emacs.d/elpa/w3m-20140107.1708/w3m-session hides
/usr/share/emacs24/site-lisp/w3m/w3m-session
~/.emacs.d/elpa/w3m-20140107.1708/w3m-mail hides
/usr/share/emacs24/site-lisp/w3m/w3m-mail
~/.emacs.d/elpa/w3m-20140107.1708/w3m-search hides
/usr/share/emacs24/site-lisp/w3m/w3m-search
~/.emacs.d/elpa/w3m-20140107.1708/w3mhack hides
/usr/share/emacs24/site-lisp/w3m/w3mhack
~/.emacs.d/elpa/w3m-20140107.1708/w3m-proc hides
/usr/share/emacs24/site-lisp/w3m/w3m-proc
~/.emacs.d/elpa/w3m-20140107.1708/w3m-tabmenu hides
/usr/share/emacs24/site-lisp/w3m/w3m-tabmenu
~/.emacs.d/elpa/w3m-20140107.1708/w3m-rss hides
/usr/share/emacs24/site-lisp/w3m/w3m-rss
~/.emacs.d/elpa/w3m-20140107.1708/mew-w3m hides
/usr/share/emacs24/site-lisp/w3m/mew-w3m
~/.emacs.d/elpa/w3m-20140107.1708/w3m hides
/usr/share/emacs24/site-lisp/w3m/w3m
~/.emacs.d/elpa/w3m-20140107.1708/w3m-image hides
/usr/share/emacs24/site-lisp/w3m/w3m-image
~/.emacs.d/elpa/w3m-20140107.1708/w3m-hist hides
/usr/share/emacs24/site-lisp/w3m/w3m-hist
~/.emacs.d/elpa/w3m-20140107.1708/w3m-lnum hides
/usr/share/emacs24/site-lisp/w3m/w3m-lnum
~/.emacs.d/elpa/w3m-20140107.1708/w3m-form hides
/usr/share/emacs24/site-lisp/w3m/w3m-form
~/.emacs.d/elpa/w3m-20140107.1708/w3m-cookie hides
/usr/share/emacs24/site-lisp/w3m/w3m-cookie
~/.emacs.d/elpa/w3m-20140107.1708/w3m-weather hides
/usr/share/emacs24/site-lisp/w3m/w3m-weather
~/.emacs.d/elpa/w3m-20140107.1708/w3m-bug hides
/usr/share/emacs24/site-lisp/w3m/w3m-bug
~/.emacs.d/elpa/w3m-20140107.1708/w3m-fb hides
/usr/share/emacs24/site-lisp/w3m/w3m-fb
~/.emacs.d/elpa/w3m-20140107.1708/w3m-antenna hides
/usr/share/emacs24/site-lisp/w3m/w3m-antenna
~/.emacs.d/elpa/w3m-20140107.1708/w3m-util hides
/usr/share/emacs24/site-lisp/w3m/w3m-util
~/.emacs.d/elpa/csv-mode-1.2/csv-mode hides
/usr/share/emacs24/site-lisp/emacs-goodies-el/csv-mode
/usr/share/emacs/24.3/site-lisp/cdargs hides
/usr/share/emacs/site-lisp/cdargs
/usr/share/emacs/24.3/site-lisp/debian-startup hides
/usr/share/emacs/site-lisp/debian-startup
/usr/share/emacs24/site-lisp/flim/md4 hides /usr/share/emacs/24.3/lisp/md4
/usr/share/emacs24/site-lisp/flim/hex-util hides
/usr/share/emacs/24.3/lisp/hex-util
/usr/share/emacs24/site-lisp/dictionaries-common/flyspell hides
/usr/share/emacs/24.3/lisp/textmodes/flyspell
/usr/share/emacs24/site-lisp/dictionaries-common/ispell hides
/usr/share/emacs/24.3/lisp/textmodes/ispell
/usr/share/emacs/site-lisp/rst hides
/usr/share/emacs/24.3/lisp/textmodes/rst
/usr/share/emacs24/site-lisp/flim/sasl-ntlm hides
/usr/share/emacs/24.3/lisp/net/sasl-ntlm
/usr/share/emacs24/site-lisp/flim/hmac-def hides
/usr/share/emacs/24.3/lisp/net/hmac-def
/usr/share/emacs24/site-lisp/flim/hmac-md5 hides
/usr/share/emacs/24.3/lisp/net/hmac-md5
/usr/share/emacs24/site-lisp/flim/sasl-cram hides
/usr/share/emacs/24.3/lisp/net/sasl-cram
/usr/share/emacs24/site-lisp/flim/ntlm hides
/usr/share/emacs/24.3/lisp/net/ntlm
/usr/share/emacs24/site-lisp/flim/sasl hides
/usr/share/emacs/24.3/lisp/net/sasl
/usr/share/emacs24/site-lisp/flim/sasl-digest hides
/usr/share/emacs/24.3/lisp/net/sasl-digest
/usr/share/emacs/24.3/lisp/emacs-lisp/ert-x hides
/usr/share/emacs/24.3/site-lisp/slime/lib/ert-x
/usr/share/emacs/24.3/lisp/emacs-lisp/cl-lib hides
/usr/share/emacs/24.3/site-lisp/slime/lib/cl-lib
/usr/share/emacs/24.3/lisp/emacs-lisp/ert hides
/usr/share/emacs/24.3/site-lisp/slime/lib/ert

Features:
(shadow sort gnus-util mail-extr emacsbug message rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils ascii rect repeat debug novice jka-compr apropos find-func
misearch multi-isearch home-end ibuf-ext ibuffer server
highlight-current-line vc-git bookmark pp saveplace desktop savehist
dired-details+ dired-details help-mode dired+ image-dired image-file
dired-x dired-aux dired uniquify winner csv-mode-autoloads
dictionary-autoloads connection-autoloads dired+-autoloads
dired-details+-autoloads dired-details-autoloads dired-ranger-autoloads
dired-hacks-utils-autoloads dired-single-autoloads
dired-sort-menu+-autoloads dired-sort-menu-autoloads dirtree-autoloads
hide-lines-autoloads igrep-autoloads link-autoloads nav-autoloads
pager-autoloads pager-default-keybindings-autoloads revive+-autoloads
revive-autoloads tree-mode-autoloads w3-autoloads w3m-autoloads info
web-autoloads dash-autoloads windata-autoloads workgroups2-autoloads
finder-inf package session woman man paren hi-lock electric delsel
debian-el debian-el-loaddefs w3m-load slime-autoloads 50magit ido
ess-toolbar ess-mouse mouseme browse-url ess-menu ess-swv ess-noweb
ess-noweb-font-lock-mode ess-bugs-l essd-els ess-sas-d ess-sas-l
ess-sas-a shell pcomplete ess-sta-d ess-sta-l cc-vars cc-defs
make-regexp ess-sp6-d ess-sp3-d ess-julia ess-r-d ess-tracebug
format-spec ess-roxy ess-help ess-developer ess-r-args eldoc ess-s-l ess
ess-inf ess-mode ess-noweb-mode ess-utils ess-custom executable
ess-compat ess-site emacs-goodies-el emacs-goodies-custom
emacs-goodies-loaddefs easy-mmode ecb cl-macs warnings edmacro kmacro
ecb-symboldef ecb-analyse ecb-compatibility ecb-winman-support
ecb-autogen autoload ecb-tod ecb-cycle ecb-eshell ecb-help ecb-jde
ecb-method-browser hideshow ecb-semantic ecb-file-browser ecb-speedbar
ecb-layout compile comint regexp-opt ansi-color tool-bar
ecb-create-layout ecb-compilation ecb-common-browser time-date assoc
speedbar sb-image dframe ecb-navigate ecb-mode-line ecb-face tree-buffer
ecb-upgrade ecb-cedet-wrapper semantic/db-mode semantic/db-find
semantic/db-ref semantic/analyze semantic/sort semantic/scope
semantic/analyze/fcn semantic/db gv eieio-base semantic/format ezimage
image semantic/tag-ls semantic/find semantic/ctxt semantic/util-modes
easymenu semantic/util semantic semantic/tag semantic/lex semantic/fw
eieio byte-opt bytecomp byte-compile cconv mode-local cedet wid-edit
ecb-util ring thingatpt dpkg-dev-el dpkg-dev-el-loaddefs devhelp
develock advice help-fns cl cl-lib advice-preload ediff-hook vc-hooks
lisp-float-type tabulated-list newcomment lisp-mode register page
menu-bar rfn-eshadow timer select mouse jit-lock font-lock syntax
facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
make-network-process dbusbind multi-tty emacs)

-- 
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

signature.asc
Description: OpenPGP digital signature