make test errata

[Martin D Kealey]

I'm trying to make sure that my modified version of Bash passes all the
tests, and in the process I'm finding some, erm, oddities.

Firstly, what seems to be a bug:

Looking in tests/printf7.sub it has
INT_MAX=$(getconf INT_MAX 2>/dev/null)
[ -z "$INT_MAX" ] && INT_MAX=2147483647         # assume 32 bits
TOOBIG=$(( $INT_MAX * 10 ))

On systems where "int" and "long" are the same underlying type, this will
simply set TOOBIG=-10, which then *doesn't* cause most of the intended
failures.

I suggest simply:

INT_MAX=$(getconf INT_MAX 2>/dev/null)
TOOBIG=${INT_MAX:-2147483647}0

Secondly, some usability issues that particularly impact new developers.
Sure most of this stuff is obvious after you've figured them out the first
time, but why force each new developer to waste that effort?

The output of “make tests” takes some getting used to, and meanwhile it
takes an unreasonably large effort to identify what comparison failed.

   - A run-on wall of lower-case-only text makes it hard to see where one
   test ends and the next begins. Some fairly minor changes to tests/run-all
   to highlight the boundaries between tests would be useful. (Even after
   getting used to hunting for lines that start with “run-”, it can still take
   a while to find exactly what's gone wrong.)
   - Keep all the test results, so that they can be reviewed in detail
   while fixing whatever problem is noted. Especially, don't have each test
   overwrite the output of the previous test; don't delete the results until
   the user runs “make tests”, or “make clean”. I suggest writing everything
   into a “build-log” subdirectory of %BUILD_DIR%.
   - Include the output of “git log -n1”
   - Having “warning:” on the start of every line, including mid-sentence,
   makes it harder to read. When there's more than warning, it makes them hard
   to split apart. (Insert blank lines between warnings; only put “warning:\t”
   on the first line of each warning, and indent the other lines with “\t”.)

I wound up modifying many test scripts so that I could *find* their output
among the large wall of text.

I'm a fan of the idea that you should be able to simply look at the output
of “make”, and copy that to replicate that part of the process. When stuff
is hidden inside environment variables, that doesn't work. This applies to
both tests and other build steps.

   - Where “make” invokes something that depends on environment variables,
   it would be helpful if it announced what those were beforehand, with a
   recipe such as “@ echo export var=$$var”.
   - Each such script (or sub-make) could have reasonable defaults for the
   environment variables it depends on, or at the least fail with a useful
   diagnostic when they're not set.
   - A large subset of the tests consist of running one script and then
   performing a diff between that and some precomputed output; these could be
   consolidated into one script that takes a test code as an argument.
   - Where there are lots of sub-tests (such as histexp[1-9].sub), make
   these into separate tests, so that when “diff” reports an issue, it's
   easier to find which one actually failed.

Where there are some “expected” differences, could these be filtered? In
particular,

   - sort the output of “trap” alphabetically, rather than leave it in
   numeric signal-number order?
   - compare whitespace-agnostic tests using “diff -w” or “diff -b”?
   - if possible compare error codes (like EINVAL) rather than
   system-dependent messages?

Could tests that depend on system facilities or build options be disabled,
or have the warnings hidden when they should not be applicable:

   - non-unicode locales are becoming less common, so in run-intl, exclude
   the 400-odd tests that depend on non-unicode multibyte encodings, or limit
   tests to those that are compatible with $LANG from the build user? (I don't
   have a terminal that can display BIG5 or SJIS, and even Latin-1 is awkward.)

conversely:

   - don't warn about needing arrays unless bash is configured without
   support for arrays
   - don't warn about needing process substitutions unless bash is
   configured without support for process substitutions or is being tested on
   a system that lacks ‘/dev/fd’ or equivalent;
   - don't warn about needing BASH_ARGV unless bash is configured without
   support for debugging

I'm prepared to submit patches to implement these if you could give me some
guidance on which of them you would accept (or not).

I have some other issues that are indirect consequences of the changes I've
made to Bash, but I'll write those up separately.

-Martin