Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Stefan Hajnoczi]

On Thu, Mar 8, 2012 at 12:28 PM, Cleber Rosa <address@hidden> wrote:
> On 03/08/2012 08:54 AM, Stefan Hajnoczi wrote:
>>
>> On Thu, Mar 8, 2012 at 11:44 AM, Stefan Hajnoczi<address@hidden>
>>  wrote:
>>>
>>> On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues<address@hidden>
>>>  wrote:
>>>>
>>>> One of our main goals is to provide useful tools for the qemu community,
>>>> since we have a good number of tests and libraries written to perform
>>>> integration/QA testing for that tool, being successfuly used by a number
>>>> of
>>>> QA teams that work on qemu. Also, we recently provided a subset of that
>>>> infrastructure to test libvirt, one of our virtualization projects of
>>>> interest.
>>>
>>> Thanks for sharing.
>>
>> One thing I should have added is that my message is about what it
>> would take for me to use autotest and contribute tests.  But now I
>> realize that you might be going for a different model:
>>
>> If you're aiming for a different model where autotest integrates
>> external test suites (i.e. tests wouldn't be written in autotest.git,
>> instead autotest.git would contain snapshots of external test suites),
>> then this proposal seems fine.  Upstream projects like QEMU would
>> develop their own test suite and it would be dropped into autotest or
>> a specific autotest instance.
>>
>> I'm not 100% sure which of these two models you're going for?
>
>
> Autotest will continue to integrate and ship with external test suites,
> even though that's an option at packaging time.
>
> But the point here is that we also want to cover the other use cases,
> which includes being able to run tests that are hosted within external
> projects, such as QEMU itself. The idea is to put two things in a state
> that's easier to be consumed by individual developers:
>
>  * The test runner
>  * The (optional) autotest API
>
> So, by making the autotest API optional, and even the language of the
> test script your own choice, you can keep you current test code, using
> your own mini-framework and still use the autotest test runner for running
> the tests and gathering the results and important system information.
>
> By improving the API, which basically means making it more visible, better
> organized and documented, we hope that users writing instrumented
> tests (using serial or ssh sessions, sending either HMP or QMP monitor
> commands, etc) will choose to use it.

It sounds like you are trying to do both the aggregation of external
test suites and provide APIs/JEOS.  Aggregation will succeed but the
API will not because aggregation undermines the effort to get folks to
use autotest.  By encouraging upstream to have their own test suites,
the API and JEOS will be taken care of upstream.  It won't make sense
for upstream developers to go to autotest when they have a test suite
in their own repository.

Upstream needs to have APIs and JEOS in order to implement their
in-tree test suites - without the ability to create VMs, interact with
the serial console, etc tests couldn't really do anything useful.
QEMU is already growing these abilities right now.

This 50/50 split seems like a compromise so that QA teams can continue
working in autotest and developers can continue working upstream.  It
doesn't change the situation we have today, that's why my first email
described a QEMU/KVM testing library (not framework).  That could
change the game because upstream tests could use the utilities that
have been developed in autotest, but without the framework.  I think
this is what we really need to do in order to grow together rather
than growing apart.  With this proposal I don't see the incentives
that will bring test development closer together, instead it seems
like a license to grow apart.

That said, I don't want to be too serious or negative about it.  It's
not the end of the world but I wanted to share my thoughts because
I've been feeling the need for better testing infrastructure (mostly
with QEMU).

Stefan