Re: [GSoC] Rewrite Hydra to be more integrated with Guix.

[Andreas Enge]

Hello,

over some lunch I have been chatting with a famous Debian developer to under-
stand a bit how Debian handles their build process. What is interesting with
their approach is that the master manages a queue of packages to be built,
and the slaves connect to fetch work when they are ready (for instance, when
they have empty resources). With our offload mechanism, the master contacts
all the build slaves to schedule the work. This is not totally different,
but I wonder whether the Debian approach is not more flexible and more geared
towards a distributed approach. (Of course, in both cases the master machine
needs to keep a list of authorised build machines, so the degree of centra-
lisation is ultimately the same.)

In the end, I think it would be interesting to not use the offload approach,
where the offloading machine sends all inputs to the build machine, which
returns the result, so that more or less the complete set of binary packages
flows along the tree of build machines to the root and the other way round.
(As I recently understood during discussions with Ludovic, currently we have
a tree of height one: hydra is the root and all build machines are the leaves;
but the offloading mechanism could transparently work with trees of bigger
height, where hydra offloads to a root of a smaller build cluster, which
itself offloads to one of its subordinate build machines. And so on. The same
thing would also work in the Debian setting, assuming that build requests
come from the leaves and are forwarded towards the root, while a work package
is handed down the other way.) Now it would be interesting if machines just
got build jobs (or made requests) and looked for the necessary inputs inside
a big magma/cloud/gnunet/ipfs/mirror network, compiled the package, and
dropped the result into the same magma. Already in our current set-up, it
would be useful if hydra need not send the build inputs, but build machines
could just fetch them from an arbitrary nginx mirror, where they are cached.

Andreas