Re: testing exceptions + bug?

[Dirk Herrmann]

> However, why are you putting all these tests in a new file called
> "exceptions"?  One would expect that this file tests the exceptions
> mechanism (catch/throw/dynamic-wind/error/etc).  What about putting
> the tests into "interp.test" or "eval.test"?  The code for dealing
> with exceptions could be factored out and put into the test-suite
> support library, "lib.scm".

I have already taken a look at factoring out exception code.  The
suggested solution is given below.  One of the ideas is to use names
'pass-if-exception' and 'expect-fail-exception' to align their names with
the 'pass-if' and 'expect-fail' macros.  Further, I classify exceptions by
'type' and by 'message', not just by message.  And, the name of the test
case is not automatically created, but has to be passed.  Only commonly
used exception types should be part of (test-suite lib), others can be
declared within the test files where they are specifically used.

The semantics of 'pass-if-exception' is to pass if the correct exception
is thrown, to fail if no exception is thrown.  If some other exception is
thrown, this will be categorized as an error.  The semantics of
'expect-fail-exception' is to pass unexpectedly if the correct exception
is thrown, to fail expectedly if no exception is thrown, and, in case of a
different exception, give an error.  The reason to react with an error
message for all other kinds of exceptions is, that otherwise programming
errors within the test cases would look like ordinary failures.  I have
realized that I have produced such a situation myself in
environments.test.  The two test cases
  leaf-environments: bound, define, ref, set!, cell: set! an undefined symbol
and
  leaf-environment based eval-environments: bound, define, ref, set!, cell: 
set! an undefined symbol
are just wrong, but this went undetected because I only checked for _some_
exception but did not make sure it was the right one :-)

One special case is not covered by these definitions, namely the case that
it is known that an exception is thrown, which is not the right one:  For
example, instead of a division by zero exception, an overflow exception is
thrown.  To be able to classify such a case as an expected failure, I
suggest to add a macro 'expect-fail-false-exception', where both the
correct exception and the false-but-currently-reported exception are
given.  Any other exception leads to an error.

If nobody objects, I will add the macros pass-if-exception and
expect-fail-exception and also provide a macro
expect-fail-false-exception.

BTW:  The test cases show, that sometimes error messages start with
capital letters and sometimes not.  Maybe we should try to be consistent
here?

Best regards,
Dirk Herrmann


---------------- Suggested addition tor (test-suite lib) -------------
;; Question of a non-macro guru:  I have unquoted string-match because I
;; wanted it to use the binding in (test-suite lib).  Otherwise every
;; other test file that tests for exceptions would have to use
;; (use-modules (ice-9 regex)).  Is this the right way of doing things?


(define exception:out-of-range
  (cons 'out-of-range "Argument out of range"))
(define exception:wrong-num-args
  (cons 'wrong-number-of-args "Wrong number of arguments"))
(define exception:wrong-type-arg
  (cons 'wrong-type-arg "Wrong type argument"))

(use-modules (ice-9 regex))

(defmacro run-test-exception (name exception expect-pass body . rest)
  `(run-test ,name ,expect-pass
     (lambda ()
       (catch (car ,exception)
         (lambda ()
           ,body ,@rest #f)
         (lambda args 
           (or (not (not (,string-match (cdr ,exception) (caddr args))))
               (apply throw args)))))))

(defmacro pass-if-exception (name exception body . rest)
  `(run-test-exception ,name ,exception #t ,body ,@rest))

(defmacro expect-fail-exception (name exception body . rest)
  `(run-test-exception ,name ,exception #f ,body ,@rest))