linking to suitesparse on OSX and relevance of buildbot "success" in building

[Carlo De Falco]

Dear all,

The continuous build of Octave on an OSX system is now up
and running and the buildbot report [1] shows it is succeeding.

Buildbot also runs 'make check' and it reports the test part as 
succeeding (even though there are 25 failing tests) [2] and none
of the tests reported as failing AFAICT pertain to suitesparse.

Buildbot does not save fntests.log among its logfiles, so what
the skipped, fail and xfail tests are cannot be checke from the
results of the automated build.

Therefore, I tried building Octave on the same system with the
same options used by the autometed build, then I ran run_octave
and __run_test_suite__ .

Here [3] and [4] you can find the files config.log and fntests.log.

To my surprise they show that although configure does not complain about
suitesparse components (e.g. cholmod or amd) not being found, and even
reports the flags used to link to them, the tests for Octave functions
depending on those libraries are skipped by 'make check'.

I'm going to post a bug about this particular issue in the tracker,
but this also raises a more fundamental question about the usefulness
of the automated tests in buildbot: what's the relevance of a reported 
"success" if there are tests that are failing and libraries that are
reported as being linked but are actually not?

how can the set-up be adjusted so that this is reported as "fail" rather
than "success"?

c.


[1] http://buildbot.octave.org:8010/builders/clang-osx
[2] http://buildbot.octave.org:8010/builders/clang-osx/builds/22
[3] http://www1.mate.polimi.it/~carlo/config.log
[4] http://www1.mate.polimi.it/~carlo/fntests.log