Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Stefan Hajnoczi]

On Fri, Mar 9, 2012 at 3:01 PM, Ademar Reis <address@hidden> wrote:
> On Fri, Mar 09, 2012 at 02:54:23PM +0000, Stefan Hajnoczi wrote:
>> On Fri, Mar 9, 2012 at 2:00 PM, Ademar Reis <address@hidden> wrote:
>> > On Fri, Mar 09, 2012 at 09:41:05AM +0000, Stefan Hajnoczi wrote:
>> >> On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis <address@hidden> wrote:
>> >> > On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
>> >> >> On 03/08/2012 04:24 PM, Ademar Reis wrote:
>> >> >> >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
>> >> >> >>On 03/08/2012 03:02 PM, Ademar Reis wrote:
>> >> >> >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
>> >> >> >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote:
>> >> >> >>>   - QE will be alienated from the qemu test effort. There will be
>> >> >> >>>     no integration between the QE efforts and the maintenance of
>> >> >> >>>     the qemu developer-level tests.
>> >> >> >>
>> >> >> >>I think we're a pretty friendly and open community :-)  There is no
>> >> >> >>reason that QE should be "alienated" unless folks are choosing not
>> >> >> >>to participate upstream.
>> >> >> >
>> >> >> >For the exact same reasons you as a developer don't want to
>> >> >> >implement tests inside autotest, QE won't want to implement tests
>> >> >> >for qemu.git. It's out of their comfort zone, just put yourself
>> >> >> >on their shoes.
>> >> >>
>> >> >> This is a really, really poor argument and I hope I don't need to go
>> >> >> into details of why.  If the primary reason for libautotest is so
>> >> >> the people writing tests for QEMU can avoid actually working with
>> >> >> the developers of QEMU...  we've got a problem.
>> >> >
>> >> > No, one of the benefits of having libautotest is to *collaborate*
>> >> > with QE. I'll explain again:
>> >> >
>> >> > - As a qemu developer, I don't want to spend my time learning and
>> >> >  getting involved in autotest, which is a complex QE project
>> >> >  (I heard this numerous times).
>> >> >
>> >> > - As a Quality Engineer, I don't want to invest my time learning
>> >> >  and getting involved into upstream qemu to test HEAD.
>> >>
>> >> I think this is the key point of the whole discussion - most of the
>> >> other topics have been distractions.  Both communities do testing but
>> >> we test different things and have different priorities.
>> >>
>> >> For me this has been the big realization from this discussion.  I felt
>> >> kvm-autotest and qemu should share tests.  I was pushing for that but
>> >> after following this thread I don't think it makes sense, here's why:
>> >>
>> >> The Quality Engineer you describe is not a QEMU upstream QE, instead
>> >> the QE has a broader and more downstream focus.  (This is why
>> >> comparisons with WebKit or other upstream projects doing testing are
>> >> not valid comparisons.)
>> >
>> > Lucas, Cleber and the others red-hatters should remembers this
>> > from my internal presentation, it was the first point I made:
>> > QE and Developers have very different goals and interests.
>> >
>> > Which is why we're pushing all these changes in autotest. We see
>> > opportunities for collaboration, but we do realize the difference.
>> >
>> > And look: Lucas and Cleber are not QE, they're developers working
>> > on the autotest framework/library/whatever. We'll need similar
>> > positions inside qemu as the test infra-structure grows.
>>
>> I don't understand this last paragraph.  If qemu.git upstream was
>> doing full-scale QE it would work fine because the differences that
>> I've described and you also have pointed out would be absent.
>>
>
> In order to have QEMU working in full "TDD Mode" (a current
> goal), I predict developers assigned to the maintenance of the
> in-house test infrastructure (qemu-test) will be needed, on
> positions similar to what Lucas and Cleber currently do with
> autotest. Only time will tell.

I agree that engineers are needed to work on testing as testing
increases upstream.

Stefan