bug#24402: should-error doesn't catch all errors

[npostavs]

Alex <agrambot@gmail.com> writes:

> I believe the following is why my previous diff doesn't work. Consider:
>
> (let ((test (make-ert-test :body (lambda () (ert-fail "failed")))))
>   (ert-run-test test))
>
> The above returns a struct/record and does not error.
>
> (let ((test (make-ert-test :body (lambda () (ert-fail "failed")))))
>   (condition-case err
>       (ert-run-test test)
>     (error "woops")))
>
> Even though ert-run-test by itself does not error, the error handler is
> ran. I believe this is because `ert--run-test-internal' binds `debugger'
> around the evaluation of the test to somehow suppress this error.

Yes, ert binds `debugger' in order to get full backtrace information
when there is an error.  This means it won't see errors caught by
condition-case.  That's good when it ignores errors caught by test code
using condition-case, but does give rise to problems.  There is some
relevant discussion in Bugs #11218 and #24617.

Espcially the suggestion in #24617 of using `signal-hook-function' to
record error info instead of using `debugger', I think doing this could
simplify things a lot.  It is definitely going to require messing around
with ert's internals though...