bug#33410: Offloaded builds can get stuck indefinitely due to network issues

[Mark H Weaver]

I just discovered that 4 out of 5 armhf build slots on Hydra have been
stuck for 24 hours, apparently after the network connections to the
build slaves were lost, possibly due to a temporary network outage.

I've seen this kind of thing happen periodically since we switched to
using guile-ssh for offloaded builds.

On Hydra I can monitor the builds and investigate when a given build
seems to be taking far too long, and I can kill those jobs to free up
the build slots.  There's no way to kill the builds from Hydra's web
interface, but I can kill them manually by logging into Hydra.

This might become a more serious problem on Berlin, as we add ARM build
slaves that are not on the same local network as Berlin itself, until
the web interface allows for this kind of monitoring and intervention.

      Mark