[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization test
From: |
Kevin Wolf |
Subject: |
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests |
Date: |
Fri, 09 Mar 2012 13:13:39 +0100 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:10.0.1) Gecko/20120209 Thunderbird/10.0.1 |
Am 09.03.2012 12:59, schrieb Anthony Liguori:
> On 03/09/2012 05:14 AM, Kevin Wolf wrote:
>> Am 09.03.2012 00:51, schrieb Ademar Reis:
>>> On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
>>>>> Plus it's not unconditional: the test runner will report tests
>>>>> SKIPPED if a dependency is not present.
>>>>
>>>> But then the tests aren't run so if most developers didn't have it
>>>> installed, and most tests were written with it, most developers
>>>> wouldn't be running most tests which defeats the purpose, no?
>>>
>>> Part of a TDD approach is to have build and test bots frequently
>>> running tests on multiple platforms with different
>>> configurations.
>>>
>>> You can't expect developers to run all tests all the time.
>>
>> I think this is one of the most important points: Not all developers
>> must run all the tests all the time.
>>
>> Actually, Anthony agreed with me when I said that developers should run
>> some sanity tests for all of qemu and maybe a few more tests for the
>> subsystems they're touching.
>
> And a small number of randomly chosen test cases.
>
> We don't want to have test cases that never run under normal circumstances or
> else they're prone to break. That's why I've talked a lot about keeping
> 'make
> check' time bound.
This sounds horrible. make check results must be reproducible, not
depend on a randomly chosen set. If you do it like this, a passed make
check means exactly _nothing_.
>> I agree that it would be bad to have a
>> autotest dependency for those basic tests that everyone should run and
>> that should take a few minutes at most.
>>
>> For the rest of test cases, however, not everyone needs to run them and
>> I think an external dependency (that is reasonably easy to satisfy) is
>> not a problem there.
>
> I'd prefer to avoid external dependencies as it just encourages people not to
> test. There may be exceptions for certain types of tests, but it should be
> an
> exception, not the rule.
>
>> A test bot would be great, but even if people just
>> run them occasionally by hand, that would already detect bugs that are
>> currently left in the code for months. If maintainers do it before
>> pushing code into master, you'll even catch everything before it goes
>> into master. This is as good as it gets.
>
> We'll integrate make check into buildbot which currently does look at
> submaintainer trees.
But make check will never be the full thing. And if you want to
integrate make check into automated testing, then choosing a random
subset of tests for each is an even worse idea than it is anyway.
I believe the only sane thing to do is to distinguish between quick
sanity tests that are run by make check, and a full test suite that is
not run by every developer, but ideally by some test bots.
Kevin
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, (continued)
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Lucas Meneghel Rodrigues, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Anthony Liguori, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Ademar Reis, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Anthony Liguori, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Ademar Reis, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Anthony Liguori, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Ademar Reis, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Stefan Hajnoczi, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Kevin Wolf, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Anthony Liguori, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests,
Kevin Wolf <=
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Anthony Liguori, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Cleber Rosa, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Anthony Liguori, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Cleber Rosa, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Anthony Liguori, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Cleber Rosa, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Lucas Meneghel Rodrigues, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Ademar Reis, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Anthony Liguori, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Paolo Bonzini, 2012/03/09