Re: makeinfo test suite

[Patrice Dumas]

On Sat, Dec 20, 2014 at 10:23:54PM +0000, Gavin Smith wrote:
> I regenerated the tests yesterday for makeinfo. Here are some notes.
> 
> I'm not sure what the difference is between the tests in "tp/t" and
> "tp/tests". Are these two completely different test suites? But
> "tp/tests/README" says that it uses init files from ../t/init.

They are different testsuites.  Tests in the t/ directory test the perl
modules, while tests in tests/ test the texi2any/makeinfo command.
In general there is no redundancy between the 2 kind of tests, except
for some tests in tests/, those that use coverage_macro.texi that tests
many elemments of Texinfo, that have already been tested in other files.
The idea is that those tests can uncover unexpected issues with
unexpected @-command connections.

Regarding files in t/init/ they aren't specific of a test suite, these
are customization files that are not 'official' customization files
(which are in tp/init/), and, indeed, are mostly used in tests.

> How to check the tests. It would be bad to regenerate the tests (i.e.
> copying the actual results to the reference results) if the actual
> results were wrong. I am adding some more to the README files about
> how to check the results.

The main way to check the tests is to look at the diff.  Sometime, for
instance if you change css (as you just did), which ends up in almost
all the html output files in tests/, you'll not be able to really check
the difference.  I can't really see a practical way to improve over
that.


I just noticed that I didn't document the 
./maintain/all_tests.sh
script.  It is what I use to manage tests in t/.  Should I document it?


Also it looks like I didn't document that there are in fact 2 kind of
tests both regarding output and input (still for tests in t/). 

Concerning input, most tests use strings in t/*.pl files.  But tests
with 'test_file' in test options, (for example cpp_lines in
t/80include.t) use input files from the t/input_files/ directory.
It is mostly for special cases that test input as file (like
cpp_lines.texi, delcomment.texi,
empty_lines_at_beginning_no_setfilename*) or when non-ascii text is
required for the tests.

Regarding output, there are also 2 cases.  In most cases, output strings
in results/*/*.pl are enough.  But sometime output files are required 
for the test.  In that case, the 'test_formats' in test output options 
will be like 'file_html'.  This is the case for some tests in
t/09indices.t, in fact for all the tests in the @file_tests array in
that file.  For those tests, in addition to a .pl file as usual in
results/*/, there is a directory with test results (in directories
prefixed by out_) and test references (in directories prefixed by res_).

> There were some tests in t that were different to regenerate. For
> example, in  t/results/indices/encoding_index_ascii, I did
> 
> cp out_html/index.html res_html/index.html

That should not have been necessary.  It should be a bug but it is
strange that it did not happen for more tests (ie, for all the file
based tests).

> to regenerate. Is there an approved way to do this? This is also
> confusing because the directory organization in
> t/results/indices/encoding_index_ascii looks like some of the
> directories in the other test suite, under "tests", again making me
> wonder what the relationship is between these two dirs.

See above, it is a case where results cannot be only in results/*/*.pl
file as strings, but actual output files are needed.  There is no strong
relationship between those and the tp/tests results, except that I used
the same convention with out_ and res_ prefixed directories.

To regenerate all the tests, I do

  ./maintain/all_tests.sh generate

but you acan see that it only calls 
  $PERL -w $file -g 
on every t/*.t files.

> I haven't attempted to run the TeX4ht or LaTeX2HTML tests yet.

The problem with these tests is that the output is not reproducible, as
it depends on the TeX4ht or LaTeX2HTML versions.

-- 
Pat