Re: [GSoC] Development of Cuirass.

[Mathieu Lirzin]

Hi,

address@hidden (Ludovic Courtès) writes:

>> 2.1 The testing infrastructure
>> ──────────────────────────────
>>
>>   Currently Cuirass is providing a small test suite consisting only of
>>   unit tests.  The integration tests are done manually with basic
>>   examples consisting in building small packages such as GNU Hello.
>>   This is not good, because actual packages take a consequent time to
>>   compile, and testing only basic examples leaves the interesting
>>   complex case out of the scope of testing.  The worst is that the
>>   result depend on the phase of the moon since every commit in Guix can
>>   potentially break the build.
>>
>>   The idea is to take inspiration from the Guix test suite by providing
>>   tiny mocked packages that will help having lightweight build
>>   specifications.  Those tests require launching another instance of the
>>   `guix-daemon' with a local temporary store.
>
> Sounds like a good idea.
>
> It would be nice to refine what it is we’re trying to test and that’s
> not addressed by unit tests.  At one end of the spectrum, I imagine we
> could be spawning a couple of GuixSD VMs connected together, one serving
> a Git repo, and the other one running Cuirass pulling from that repo (a
> system test).
>
> At the middle of the spectrum is something like you describe, I think:
> run Cuirass directly upon “make check”, though there are complications:
> what repo do we pull from, do we really want to build packages and if
> not how to we mock builds, etc.

The idea I had was to create fake vcs repositories programmatically.  I
imagine that it is possible to associate some dummy build script that we
can deterministically make fail.

> There are several components we’d like to test more closely IMO: SCM
> polling, evaluation, and job scheduling.  Perhaps some of that can also
> be achieved through unit tests?

Yeah, that would be better.

>> 2.2 HTTP API
>> ────────────
>>
>>   Since commit [12d71ee098c3eb84a9a5dc2419bd37a3b9e55dd2] Cuirass has a
>>   web server which is running in a separate thread of the `cuirass'
>>   process.  However the resources it provides are pretty limited since
>>   you can only query the lists of specifications currently watched.
>>   This is done with the `/specifications' or `/jobsets' resource names.
>>   "Jobsets" is the terminology used by Hydra and is used here for
>>   compatibility with its HTTP API.
>
> For a start, I think we should implement all or a subset of the Hydra
> API, as-is (it’s currently used by Emacs-Guix):
>
>   https://github.com/NixOS/hydra/blob/master/src/lib/Hydra/Controller/API.pm
>
> A large part of it is about monitoring the status of Hydra: queued
> builds, recently-completed builds, jobs, etc.  That is the most pressing
> need, I think.

OK.

>>   This roadmap lacks details about the concrete tasks and its associated
>>   timeline.  However I prefer having some feedback about this proposal
>>   first, before providing a complete roadmap.
>>
>>   • From May 30 to June 26, I will work on the testing infrastructure,.
>>   • From June 27 to August 29, I will work on the HTTP API.
>
> You may find that you’ll want to interleave tasks.  I don’t know about
> you, but sometimes I get bored if I keep working on the same task or
> domain for too long, so switching to something else for a while can be a
> relief.  :-)

I don't know, I have been rarely confronted to that issue personnaly
(more the opposite: too much context switching).  But I will
preventively adapt my roadmap.

> There are a couple of isolated tasks that come to mind, which do not
> really fit in the proposal but are worth looking into as time permits:
>
>   • improved error reporting upon evaluation failures and so on;
>
>   • improved error recovery;

I totally forgot to precise that the error handling issue is exactly the
reason I think we need a better test infrastructure for Cuirass.  Being
able to reproduce various errors or failues allows a better confidence
in the error being properly handled.  I find it hard to just imagine
and write the appropriate handler.

>   • use of Guile-Git instead of Git.

I Will add this to the plan.

Thanks.

-- 
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37