Re: Why isn't ert-buffer.el part of Emacs?

[Thorsten Jolitz]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I recently discovered 'ert-buffer.el' and find it very useful.  It is
>> introduced and discussed here:
>> I wonder why such a generally useful library isn't included in Emacs (I
>> don't even find it in the package repos)?
>
> I vaguely remember hearing about ert-buffer.el at some point.  I don't
> have enough time to investigate its usefulness myself, so I can only
> judge it based on how much it's used.

My question had probably two parts:

 - find out if I miss something in core Emacs that gives me the same
   functionality as ert-buffer.el?

 - raise some attention to ert-buffer.el 's usefullness

> One way to show its usefulness would be to write a patch for Emacs's
> test/automated subdirectory showing how it lets us simplify those tests.

Thats probably a bit over my head, and I don't have the time right now. 

> I think if such a library is useful, then it belongs in Emacs's core
> rather than in GNU ELPA, since we'd probably want to use it for Emacs's
> own tests.

I did write a little extension to it called ert-buffer-report.el (not
yet published) that prints the test results into an Org file, so I can
demonstrate how the library is usefull for me for testing the conversion
between programming modes and org-mode via outorg.el.

When testing this kind of conversion, often my MWEs (mininal working
example) pass the tests, but when I try one of the huge complex
elisp/org documents around (I use e.g. Fabrice Nielson's giant former
init.el and Bernt Hansen's big org-mode.org for this) I get
failures. Easily creating 'black-box' tests for these really big
real-world files/buffers (not just MWEs) with ert-buffer.el comes in
handy.

Here I test a MWE that does expose a bug (which I found by testing big
real-world files). Before running the tests, I do 

,--------------------------------------------------
| M-x ert-buffer-report-toggle-insert-buffer-string
`--------------------------------------------------

so the (short) MWE content BEFORE and AFTER the test is included in the
report. 

1. WITH BUG:

,-------------------------------------------------------
| After 
|  - converting from elisp [BEFORE] to org
|  - running an org-command and storing buffer-undo-tree
|  - converting back from org to elisp
| and then 
|  - converting from elisp to org (again)
|  - undoing changes using stored buffer-undo-tree 
|  - converting back from org to elisp [AFTER]
`-------------------------------------------------------

I get this org-mode report:
 - Point moved
 - content length increased
 - there are content DIFFS between BEFORE and AFTER
thus the test failed due to unwanted conversion side-effects

#########################################################################
* ERT Test Report <2014-06-17 Di 11:42>
** Point, Mark and Content Lenght
 - Point position :: 534 -> 38
 - Mark position :: nil -> nil
 - Content length :: 533 -> 583
** Content DIFF
#+begin_quote
15,19c15,16
< (add-hook 'org-mode-hook (lambda () (abbrev-mode 1)))
< 
< (setq org-todo-keywords
<       (quote ((sequence "TODO(t)" "NEXT(n)" "|" "DONE(d!/!)")
<               (sequence "WAITING(w@/!)" "HOLD(h@/!)" "|" "CANCELLED(c@/!)" 
"PHONE"))))
---
20a18,21
> ;; #+begin_src emacs-lisp
> ;; (add-hook 'org-mode-hook (lambda () (abbrev-mode 1)))
> ;; (setq org-todo-keywords
> ;;       (quote ((sequence "TODO(t)" "NEXT(n)" "|" "DONE(d!/!)")
> ;;               (sequence "WAITING(w@/!)" "HOLD(h@/!)" "|" "CANCELLED(c@/!)" 
> "PHONE"))))
> ;; #+end_src

#+end_quote
** Buffer Strings
*** Buffer String BEFORE
#+begin_quote

;; *** General

;; **** Configuration

(setq org-indent-mode-turns-on-hiding-stars nil
      org-replace-disputed-keys t
      org-use-speed-commands t)

(add-hook 'org-mode-hook
          (lambda ()
            (local-set-key (kbd "C-c M-o") 'tj/mail-subtree))
          'append)

;; Enable abbrev-mode
(add-hook 'org-mode-hook (lambda () (abbrev-mode 1)))

(setq org-todo-keywords
      (quote ((sequence "TODO(t)" "NEXT(n)" "|" "DONE(d!/!)")
              (sequence "WAITING(w@/!)" "HOLD(h@/!)" "|" "CANCELLED(c@/!)" 
"PHONE"))))

^@
#+end_quote
*** Buffer String AFTER
#+begin_quote

;; *** General

;; **** Configuration^@

(setq org-indent-mode-turns-on-hiding-stars nil
      org-replace-disputed-keys t
      org-use-speed-commands t)

(add-hook 'org-mode-hook
          (lambda ()
            (local-set-key (kbd "C-c M-o") 'tj/mail-subtree))
          'append)

;; Enable abbrev-mode
;; #+begin_src emacs-lisp
;; (add-hook 'org-mode-hook (lambda () (abbrev-mode 1)))

;; (setq org-todo-keywords
;;       (quote ((sequence "TODO(t)" "NEXT(n)" "|" "DONE(d!/!)")
;;               (sequence "WAITING(w@/!)" "HOLD(h@/!)" "|" "CANCELLED(c@/!)" 
"PHONE"))))
;; #+end_src

#+end_quote
#########################################################################

It seems that, when converting back from org-mode to elisp, my regexp
that matches emacs-lisp code-blocks in org-mode did not match the block
that contains @ symbols. This seemed strange for me, because I use

,--------------------------------------------------------
| "^#\\+begin_src[[:space:]]+emacs-lisp[^^@]*\n#\\+end_src"
`--------------------------------------------------------

and that negated zero-byte in [^^@]* should really match everything
inside the block delimiters. After some experimenting I figured out that
my regexp was buggy, since I simply inserted ^^@ as three normal chars 

,------
| ^ ^ @
`------

while I should have used  

,----------
| ^ C-q C-@
`----------

to actually use the zero-byte. 


2. WITHOUT BUG:

Repeating this test, this time omitting the buffer strings by calling 

,--------------------------------------------------
| M-x ert-buffer-report-toggle-insert-buffer-string
`--------------------------------------------------

again, I get:


##########################################
* ERT Test Report <2014-06-17 Di 11:59>
** Point, Mark and Content Lenght
 - Point position :: 534 -> 38
 - Mark position :: nil -> nil
 - Content length :: 533 -> 532
** Content DIFF
#+begin_quote
20d19
< 

#+end_quote
** Buffer Strings
*** Buffer String BEFORE
   [buffer-string omitted]
*** Buffer String AFTER
   [buffer-string omitted]
##########################################

Point moved again, what makes the test fail, but isn't relevant in this
case. Now content-length changed by only 1 char, and the DIFF looks like
this would be a newline \n. 

Lets print the content strings again to get a clearer picture:


#################################################################
* ERT Test Report <2014-06-17 Di 12:04>
** Point, Mark and Content Lenght
 - Point position :: 534 -> 38
 - Mark position :: nil -> nil
 - Content length :: 533 -> 532
** Content DIFF
#+begin_quote
20d19
< 

#+end_quote
** Buffer Strings
*** Buffer String BEFORE
#+begin_quote

;; *** General

;; **** Configuration

(setq org-indent-mode-turns-on-hiding-stars nil
      org-replace-disputed-keys t
      org-use-speed-commands t)

(add-hook 'org-mode-hook
          (lambda ()
            (local-set-key (kbd "C-c M-o") 'tj/mail-subtree))
          'append)

;; Enable abbrev-mode
(add-hook 'org-mode-hook (lambda () (abbrev-mode 1)))

(setq org-todo-keywords
      (quote ((sequence "TODO(t)" "NEXT(n)" "|" "DONE(d!/!)")
              (sequence "WAITING(w@/!)" "HOLD(h@/!)" "|" "CANCELLED(c@/!)" 
"PHONE"))))

^@
#+end_quote
*** Buffer String AFTER
#+begin_quote

;; *** General

;; **** Configuration^@

(setq org-indent-mode-turns-on-hiding-stars nil
      org-replace-disputed-keys t
      org-use-speed-commands t)

(add-hook 'org-mode-hook
          (lambda ()
            (local-set-key (kbd "C-c M-o") 'tj/mail-subtree))
          'append)

;; Enable abbrev-mode
(add-hook 'org-mode-hook (lambda () (abbrev-mode 1)))

(setq org-todo-keywords
      (quote ((sequence "TODO(t)" "NEXT(n)" "|" "DONE(d!/!)")
              (sequence "WAITING(w@/!)" "HOLD(h@/!)" "|" "CANCELLED(c@/!)" 
"PHONE"))))

#+end_quote
#############################################################################


So point moved from end-of-buffer (BEFORE)

,--------------------------------------------------------------------------
|  (sequence "WAITING(w@/!)" "HOLD(h@/!)" "|" "CANCELLED(c@/!)" "PHONE"))))
| 
| ^@
`--------------------------------------------------------------------------

to 

,---------------------
| **** Configuration^@
`---------------------

(AFTER) and somehow that last empty line where point was disappeared. So
conversion works (almost) fine already ...


Note that I get all the info about point, mark, buffer-content and
buffer-string BEFORE and AFTER for free from ert-buffer.el. Although I
had to write some code to setup the environment for testing outorg
(outorg-test.el), the actual ERT test using ert-buffer.el looks like
this

,-----------------------------------------------------
| (ert-deftest outorg-test-conversion ()
|   "Test outorg conversion to and from Org.[...]"
|   (let ((curr-buf-initial-state
|          (with-current-buffer "*outorg-test-buffer*"
|            (ert-Buf-from-buffer))))
|     (should
|      (ert-equal-buffer
|       (outorg-test-cmd)
|       curr-buf-initial-state
|       t))))
`-----------------------------------------------------

which is short enough I think ...

-- 
cheers,
Thorsten