[PATCH 0/7] RFC: gitlab: enable use of ccache in FreeBSD / macOS Cirrus CI jobs

[Daniel P . Berrangé]

We've seen that we exhaust the Cirrus CI free credits for the QEMU
account, especially during the end of a release cycle.

It is hard to cut down the CI credits burn beyond its current level
without dropping features from the build configuration, which harms
CI coverage.

This proposes two things

 * First, when CI credits are exhausted we stop treating Cirrus CI
   failures as fatal. Instead the FreeBSD/macOS jobs get marked as
   "warning" meaning they won't break the whole pipeline.

 * Enable the use of ccache in the build process which would reduce
   the wallclock time in jobs, when there is a good cache hit rate.

...alot of heavy lifting is done by the last part of that sentence
above, meaning this isn't a guaranteed win. When ccache takes a
cache miss, compile time is worse than without ccache. Also we have
a time penalty of uploading/downloading the cache at start/end of
the job.

Observed timings over a number of runs:

 * No ccache (ie today)

    *   macOS: 6m00 build,  8m00 total
    *   macOS: 6m30 build,  8m20 total
    * FreeBSD: 8m45 build, 14m50 total
    * FreeBSD: 9m00 build, 15m30 total

   In macOS we see approx 2m for brew install tasks
   In FreeBSD we see approx 5m30-6m for pkg install tasks

 * ccache: cold cache (ie 0% hit rate)

    *   macOS: 8m15 build, 12m00 total
    *   macOS: 7m55 build, 10m03 total
    * FreeBSD: 9m35 build, 16m00 total
    * FreeBSD: 9m23 build, 15m15 total

   In macOS we see the compile time increase by 1m30-2m
   In FreeBSD we see the compile time increase 0m30-1m

 * ccache: hot cache (ie 99% hit rate)

    *   macOS: 5m25 build, 8m37 total
    *   macOS: 3m28 build, 5m35 total
    *   macOS: 3m40 build, 6m00 total
    * FreeBSD: 5m31 build, 12m00 total
    * FreeBSD: 5m25 build, 11m50 total
    * FreeBSD: 5m20 build, 11m00 total

   In macOS we see compile time decrease 0m30-3m
   In FreeBSD we see compile time decrease 3m

   Most of the "build" time remaining here is actually
   running of unit tests & qtests.

   There is some non-deterministic upload/download overhead
   between 0m05-0m30.

What this tells us:

  * ccache with cold cache is always slower than today
  * ccache with hot cache is usually better than today

The $1 million question is what our cache hit rate will look like
across real world pull requests ?

Also any slowness in homebrew / ports can impact our pkg install
times non-deterministically.

It is very hard to say for sure if this is a win, without actually
trying it over a release cycle and seeing how our CI credits usage
varies over 4 months. Even that will be hard, as we QEMU is a
moving target adding features. eg I wonder what Rust would do to
build times, if anything measurable ?

So our options

 * Trial ccache with Cirrus CI
 * Cull more features from macOS/FreeBSD build config
 * Reduce 'make check' coverage for macOS/FreeBSD
 * Get our own CI private runners for FreeBSD and use Cirrus
   only for macOS
 * <Your idea here>

Daniel P. Berrangé (7):
  gitlab: don't fail cirrus CI jobs when credits are exhausted
  gitlab: use new(ish) cirrus-vars command for creating config
  gitlab: clean packages in cirrus tasks
  gitlab: purge build files from cirrus CI jobs
  gitlab: enable ccache with Cirrus CI
  tests: update libvirt-ci submodule for newer ccache
  gitlab: force ccache to validate compiler version

 .gitlab-ci.d/cirrus.yml             | 30 +++++++++--------------------
 .gitlab-ci.d/cirrus/build.yml       | 14 +++++++++++++-
 .gitlab-ci.d/cirrus/freebsd-14.vars |  2 +-
 tests/lcitool/libvirt-ci            |  2 +-
 tests/vm/generated/freebsd.json     |  2 +-
 5 files changed, 25 insertions(+), 25 deletions(-)

-- 
2.46.0