Re: [Chicken-users] Unit testing

[Alejandro Forero Cuervo]

> Please elaborate on "specify limits for the execution".

Well, I was thinking of setrlimit(2).  Perhaps also of setreuid(2).
Things like "this expresion should never take more than 5 seconds to
evaluate and no more than 1MB of memory" would be good.

> As far as "crash" proof goes, that will require something that traps
> memory exceptions, etc. None of the existing 3 support this.  Also
> requires a test results storage system that is incremental,
> transactional, & probably external. "test-infrastructure" constructs
> its' results in a functional manner, "check" is incremental but not
> safe, & "testeez" is incremental but not safe.

I was thinking of just forking a separate process that runs the test
and passes the result back over a pipe(2).

> I do maintain "test-infrastructure" but am not the author. Think I
> will make the necessary thread-safety changes.

I'm afraid using threads wouldn't be safe enough.  The tests would
probably test C code from the eggs.  If I understand how threads in
Chicken work, a bad low-level error in the tested code could easily
crash the whole testing process.

I guess my ideal implementation for defining tests would be along
these lines:

    (test-compare
      (car (cons 1 2)) ; expresion
      1                ; expected value
      equal?)          ; compare function

    (test-assert
      (cons 1 2)
      pair?)

We would have lots of files with all these tests (one for each egg).
And then we would reference them from another file as this:

    (run-tests
      "lists.scm"        ; file
      "Basic list tests" ; description of tests
      1                  ; max memory tests for each test (MB)
      5)                 ; max number of seconds for each test

This would generate a report for each test defined in lists.scm.
Each test would be run as its own process.  Ideally, the process would
set its UID to a specific user right before running the code.  Also, I
think the test files should allow defining symbols that the tests can
use but that different files shouldn't be able to interfere with each
other (so I would probably also fork a process before actually running
the code from each file).  I also think run-tests should try to run
the tests both in compiled as in interpreted mode.

So, of the top of my head, that's it all.

Alejo.
http://azul.freaks-unidos.net/