Re: [Qemu-devel] KVM call minutes for Apr 5

[Stefan Hajnoczi]

On Fri, Apr 08, 2011 at 09:58:22AM -0300, Lucas Meneghel Rodrigues wrote:
> On Thu, 2011-04-07 at 11:03 +0100, Stefan Hajnoczi wrote:
> > On Tue, Apr 5, 2011 at 6:37 PM, Lucas Meneghel Rodrigues <address@hidden> 
> > wrote:
> > >> Perhaps kvm-autotest is a good platform for the automated testing of
> > >> ARM TCG.  Paul is CCed, I recently saw the Jenkins qemu build and boot
> > >> tests he has set up.  Lucas, do you have ideas on how these efforts
> > >> can work together to bring testing to upstream QEMU?
> > >> http://validation.linaro.org/jenkins/job/qemu-boot-images/
> > >
> > > I heard about jenkins before and it is indeed a nice project. What they
> > > do here, from what I could assess browsing at the webpage you provided
> > > is:
> > >
> > > 1) Build qemu.git every time there are commits
> > > 2) Boot pre-made 'pristine' images, one is a lenny arm image and the
> > > other is a linaro arm image.
> > >
> > > It is possible to do the same with kvm autotest, just a matter of not
> > > performing guest install tests and executing only the boot tests with
> > > pre-made images. What jenkins does here is a even quicker and shorter
> > > version of our sanity jobs.
> > >
> > > About how we can work together, I thought about some possibilities:
> > >
> > > 1) Modify the jenkins test step to execute a kvm autotest job after the
> > > build, with the stripped down test set. We might gain some extra debug
> > > info, that the current test step does not seem to provide
> > > 2) Do the normal test step and if that succeeds, trigger a kvm autotest
> > > job that does more comprehensive testing, such as migration, time drift,
> > > block layer, etc
> > >
> > > The funny thing is that KVM autotest has infrastructure to do the same
> > > as jenkins does, but jenkins is highly streamlined for the buildbot use
> > > case (continuous build and integration), and I see that as a very nice
> > > advantage. So I'd rather keep use jenkins and have kvm autotest plugged
> > > into it conveniently.
> > 
> > That sounds good.  I think the benefit of working together is that
> > different entities (Linaro, Red Hat, etc) can contribute QEMU tests
> > into a single place.  That testing can then cover to both upstream and
> > downstream to prevent breakage.
> > 
> > So kvm-autotest can run in single job mode and kicked off from jenkins
> > or buildbot?
> > 
> > It sounds like kvm-autotest has or needs its own cron, result
> > archiving, etc infrastructure.  Does it make sense to use a harness
> > like jenkins or buildbot instead and focus kvm-autotest purely as a
> > testing framework?
> 
> In the context that there are already jenkins/buildbot servers running
> for qemu, having only the KVM testing part (autotest client + kvm test)
> is a possibility, to make things easier to plug and work with what is
> already deployed.
> 
> However, not possible to focus KVM autotest as a 'test framework'. What
> we call KVM autotest is in actuality, a client test of autotest.
> Autotest is a generic, large collection of programs and libraries
> targeted at peforming automated testing on the linux platform, it was
> developed to test the linux kernel itself, and it is used to do
> precisely that. Look at test.kernel.org. All those tests are executed by
> autotest.
> 
> So, autotest is much more than KVM testing, and I am one of the autotest
> maintainers, so I am commited to work on all parts of that stack.
> Several testing projects urelated to KVM use our code, and our
> harnessing and infrastructure is already pretty good, we'll keep to
> develop it.
> 
> The whole thing was designed in a modular way, so it's doable to use
> parts of it (such as the autotest client and the KVM test) and integrate
> with stuff such as jenkins and buildbot, and if people need and want to
> do that, awesome. But we are going to continue develop autotest as a
> whole framework/automation utilities/API, while developing the KVM test.

I wasn't aware of the scope of autotest and your involvement.  I need to
look into autotest more :).

Stefan