Re: [Qemu-devel] KVM call minutes for Apr 5

[Stefan Hajnoczi]

On Tue, Apr 5, 2011 at 6:37 PM, Lucas Meneghel Rodrigues <address@hidden> wrote:

Thanks for your detailed response!

> On Tue, 2011-04-05 at 16:29 +0100, Stefan Hajnoczi wrote:
>> * Public notifications of breakage, qemu.git/master failures to
>> qemu-devel mailing list.
>
> ^ The challenge is to get enough data to determine what is a new
> breakage from a known issue, mainly. More related to have historical
> data from test results than anything else, IMO.

I agree.  Does kvm-autotest currently archive test results?

>> * A one-time contributor can get their code tested.  No requirement to
>> set up a server because contributors may not have the resources.
>
> Coming back to the point that many colleagues made: We need a sort of
> 'make test' on the qemu trees that would fetch autotest and could setup
> basic tests that people could run, maybe suggest test sets...
>
> The problem I see is, getting guests up and running using configs that
> actually matter is not trivial (there are things such as ensuring that
> all auxiliary utilities are installed in a distro agnostic fashion,
> having bridges and DHCP server setup on possibly a disconnected work
> laptop, and stuff).
>
> So, having a 'no brains involved at all' setup is quite a challenge,
> suggestions welcome. Also, downloading isos, waiting for guests to
> install and run thorough tests won't be fast. So J. Random Developer
> might not bother to run tests even if we can provide a fool proof,
> perfectly automated setup, because it'd take a long time at first to get
> the tests run. This is also a challenge.

I'm actually starting to think that there is no one-size-fits-all solution.

Developers need "make check"-type unit tests for various QEMU
subsystems.  kvm-autotest could also run these unit tests as part of
its execution.

Then there are end-to-end acceptance tests.  They simply require
storage, network, and time resources and there's no way around that.
These tests are more suited to centralized testing infrastructure that
periodically tests qemu.git.

On the community call I was trying to see if there is a "lightweight"
version of kvm-autotest that could be merged into qemu.git.  But now I
think that this isn't realistic and it would be better to grow unit
tests in qemu.git while covering it with kvm-autotest for acceptance
testing.

>> Perhaps kvm-autotest is a good platform for the automated testing of
>> ARM TCG.  Paul is CCed, I recently saw the Jenkins qemu build and boot
>> tests he has set up.  Lucas, do you have ideas on how these efforts
>> can work together to bring testing to upstream QEMU?
>> http://validation.linaro.org/jenkins/job/qemu-boot-images/
>
> I heard about jenkins before and it is indeed a nice project. What they
> do here, from what I could assess browsing at the webpage you provided
> is:
>
> 1) Build qemu.git every time there are commits
> 2) Boot pre-made 'pristine' images, one is a lenny arm image and the
> other is a linaro arm image.
>
> It is possible to do the same with kvm autotest, just a matter of not
> performing guest install tests and executing only the boot tests with
> pre-made images. What jenkins does here is a even quicker and shorter
> version of our sanity jobs.
>
> About how we can work together, I thought about some possibilities:
>
> 1) Modify the jenkins test step to execute a kvm autotest job after the
> build, with the stripped down test set. We might gain some extra debug
> info, that the current test step does not seem to provide
> 2) Do the normal test step and if that succeeds, trigger a kvm autotest
> job that does more comprehensive testing, such as migration, time drift,
> block layer, etc
>
> The funny thing is that KVM autotest has infrastructure to do the same
> as jenkins does, but jenkins is highly streamlined for the buildbot use
> case (continuous build and integration), and I see that as a very nice
> advantage. So I'd rather keep use jenkins and have kvm autotest plugged
> into it conveniently.

That sounds good.  I think the benefit of working together is that
different entities (Linaro, Red Hat, etc) can contribute QEMU tests
into a single place.  That testing can then cover to both upstream and
downstream to prevent breakage.

So kvm-autotest can run in single job mode and kicked off from jenkins
or buildbot?

It sounds like kvm-autotest has or needs its own cron, result
archiving, etc infrastructure.  Does it make sense to use a harness
like jenkins or buildbot instead and focus kvm-autotest purely as a
testing framework?

Stefan