[LOCAL-CI 0/3] Add support for local automated testing

[Glenn Washburn]

This patch series aims to make automated testing of all supported targets easy.
Not all targets are actually tested for a variety of reasons, but this series
covers a significant portion of the target space and should make it easy to
add more targets as we figure out how do run their tests. All supported targets
(that I know of) are build tested.

The main work horse is the scripts/local-tester.sh script which ties everything
together. It is very configurable, but defaults to building and testing as much
as it can. To start it just run "./scripts/local-tester.sh" from the root of the
repository. By default a directory named 'grub-tests' in the current working
directory is created and everything is put in there. See the beginning of
./scripts/ci/function.sh for available environment variables that can be used
to configure how you want it to run.

The file ./scripts/ci/function.sh has a bunch of functions most of which are
for stages in the automated testing. It is intended to be sourced by config
files for other CI systems so that they can reuse this code.

One might ask, why not just have all CI systems use local-tester.sh and put
everything in there? The issue is that currently local-tester.sh does not do
things in parallel (the make subprocess can be made more parallel with the
JOBS environment variable). So in many CI systems, one could have all targets
building and testing at the same time, which local-tester.sh only does this
serially. This makes local-tester.sh a lot slower than it needs to be. Also
other CI systems allow the CI pipeline to e broken into many stages, each of
which could be run on a different machine, with the ability to cache the
output of certain stages.

These scripts have been written and tested on debian based systems, specifically
the reference system, Debian 11. Some of the stages are debian or debian-
derivative specific, such as the package install stage which uses apt and dpkg.
Most of the stages, I believe, are fairly distro agnostic and the ones that
aren't should be able to be adapted for a specific distro fairly easily.

Also, this patch series is meant to be used on top of the grub-shell patch
series already submitted to the list. It will run without that series, but
some of the features may not work or work as well. Noteably, the QEMU tests
for targets i386-efi, arm-efi and arm64-efi will fail.

Of particular note, there are some knobs that can provide a lot debugging
output and save the intermediate files of failed tests for later analysis.

local-tester.sh will try to download and install all packages it needs to
function. Obviously, this will not work when not running as a privileged user.
A further patch series is intended, which will add support for running the
system successfully completely as an unprivileged user. If local-tester.sh is
run as an unprivileged user, it will skip running of privileged commands, like
the package installer. This means it can continue past the package install
phase, but it assumes that the needed packages are already installed.

Patch 1: Add general scripts for automated building/testing.
Patch 2: Add automated building/testing stages and respective configuration
  variables.
Patch 3: Add consumer of those stages, local-tester.sh, for running the
  automated building/testing locally.

I'm not particularly in love with the name "local-tester.sh", so suggestions on
alternative names are welcome. The same goes for the output, which I think
could look better.

Glenn

Glenn Washburn (3):
  scripts: Add general scripts to aid automated testing
  scripts: Add functions for CI stages and default variables to
    functions.sh
  scripts: Add local-tester.sh script to run local CI tests

 scripts/ci/build.sh           |  67 +++
 scripts/ci/functions.local.sh |  37 ++
 scripts/ci/functions.sh       | 930 ++++++++++++++++++++++++++++++++++
 scripts/ci/make-images.sh     |  86 ++++
 scripts/ci/process-tests.sh   | 111 ++++
 scripts/ci/test.sh            | 120 +++++
 scripts/local-tester.sh       |  39 ++
 7 files changed, 1390 insertions(+)
 create mode 100755 scripts/ci/build.sh
 create mode 100644 scripts/ci/functions.local.sh
 create mode 100644 scripts/ci/functions.sh
 create mode 100755 scripts/ci/make-images.sh
 create mode 100755 scripts/ci/process-tests.sh
 create mode 100755 scripts/ci/test.sh
 create mode 100755 scripts/local-tester.sh

-- 
2.27.0