Re: [Orgmode] [babel] Babel as a test harness?

[Nathan Neff]

On Fri, Mar 19, 2010 at 7:45 PM, Dan Davison <address@hidden> wrote:

Nathan Neff <address@hidden> writes:

> I think it would be cool to use Org-babel as a simple test harness.
>
> I'd like to have an org file, with various sections that demonstrate how to do
> something in Groovy.  I'd like to be able to run all the code in the org file
> and make sure
> they all run successfully (return code 0).
>
> In Groovy, the "assert" function will exit with a non-zero code if it fails. 
> How
> would I use org-babel to generate a "summary" table with the name of
> each patch of code and whether or not it succeeded?

Like below? This is all Eric's doing. For quite a while now org-babel
has used a table like this to validate itself. It took me a while to
understand it, but basically Eric designed a special function (actually,
a macro) called sbe (source block evaluate) to be used in table formulas
to call org-babel source blocks. Use C-u C-c C-c to update the table.

* Tests
| functionality  | block        | arg | expected | results              | pass                                       |
|----------------+--------------+-----+----------+----------------------+--------------------------------------------|
| simple regexp  | simple_regex |     |          |                      | pass                                       |
| regexp with or | regex_or     |     |          | A pretend problem... | expected "" but was "A pretend problem..." |
#+TBLFM: $5='(if (= (length $3) 1) (sbe $2 (n $3)) (sbe $2)) :: $6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))

* Code blocks for tests

** Here's how to match "foo"
#+source: simple_regex
#+begin_src groovy
assert "foo" =~ /foo/
#+end_src

** Here's how to match "bar" or "baz"
#+source: regex_or
#+begin_src groovy
assert "bar" =~ "ba(z|r)"

println "A pretend problem"
#+end_src

You can put any arguments to your source blocks in the arg column. See

http://orgmode.org/worg/org-contrib/babel/intro.php#spreadsheet

(It often makes most sense to clone Worg and view the org files
themseleves)

And for a more complex example, including how to pass arguments to
source blocks to sbe, see our full test suite in the file
development.org in the devel repo

http://github.com/eschulte/babel-dev/

Dan

This is really cool Dan!  I simplified the test table -- it runs the test in column 1, then
prints the output to the second column, and puts "pass" or "fail" in the 3rd column
if the 2nd column is blank.

* Tests, Simpler

Output from each source block should be nothing.
| block        | output                                                        | result |
|--------------+---------------------------------------------------------------+--------|
| simple_regex |                                                               | pass   |
| regex_or     |                                                               | pass   |
| regex_fail   | Caught: java.lang.AssertionError: _expression_: (bar =~ foo)... | fail   |
#+TBLFM: $2='(sbe $1) :: $3='(if (string= $2 "") "pass" "fail")


* Code blocks for tests
** Here's how to match "foo"
#+source: simple_regex
#+begin_src groovy results: output
assert "foo" =~ /foo/
#+end_src

** Here's how to match "bar" or "baz"
#+source: regex_or
#+begin_src groovy
assert "bar" =~ "ba(z|r)"
#+end_src

** A failing test
#+source: regex_fail
#+begin_src groovy
assert "bar" =~ "foo"
#+end_src