Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[John Snow]

On 05/08/2018 05:56 AM, Stefan Hajnoczi wrote:
> On Thu, Apr 26, 2018 at 10:57:55AM -0300, Eduardo Habkost wrote:
>> (Starting a new thread, for more visibility)
>>
>> (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
>> devices with all machines, not only with "none")
>>
>> On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
>>> Thomas Huth <address@hidden> writes:
>>>> On 17.04.2018 14:12, Markus Armbruster wrote:
>>>>> Thomas Huth <address@hidden> writes:
>>>>>
>>>>>> Many device introspection crashes only happen if you are using a
>>>>>> certain machine, e.g.:
>>>>>>
>>>>>> $ ppc-softmmu/qemu-system-ppc -S -M ref405ep,accel=qtest -qmp stdio
>>>>>> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 11, "major": 2},
>>>>>>  "package": "build-all"}, "capabilities": []}}
>>>>>> { 'execute': 'qmp_capabilities' }
>>>>>> {"return": {}}
>>>>>> { 'execute': 'device-list-properties',
>>>>>>   'arguments': {'typename': 'macio-newworld'}}
>>>>>> Unexpected error in qemu_chr_fe_init() at chardev/char-fe.c:222:
>>>>>> Device 'serial0' is in use
>>>>>> Aborted (core dumped)
>>>>>>
>>>>>> To be able to catch these problems, let's extend the device-introspect
>>>>>> test to check the devices on all machine types. Since this is a rather
>>>>>> slow operation, the test is only run in "SPEED=slow" mode.
>>>>>
>>>>> If the device works with one machine type, it has a decent chance to
>>>>> work with others, too.  Thus, testing each device with every machine
>>>>> type is overkill.  I appreciate having overkill as an option :)
>>>>>
>>>>> What I'd like to see for a quick "make check" is testing each device
>>>>> once.  That should flush out most bugs.  
>>>>
>>>> That's already done with the "none" machine.
>>>
>>> I was too terse.  We test each device with -machine none for every
>>> target.  Fine if that's quick enough.  If not, we might want to reduce
>>> redundancy there.
>>>
>>> Actually, a worse offender in the "waste everybody's time via redunancy"
>>> department could be qom-test.
>>>
>>>> Anyway, do you think my patch here is useful and has a chance of getting
>>>> included? I.e. shall I re-spin this as a non-RFC patch? Or shall we
>>>> rather wait for Eduardo's python-based tests to get included into the
>>>> repository?
>>>
>>> I don't mind having make check SPEED=slow run more extensive tests.
>>> Assuming we actually run them at least once in a while, which seems
>>> doubtful.
>>
>> We probably don't do that, but we really must be running a more
>> extensive (and slower) test set at least once before every
>> release.
>>
>> Maybe some people are running SPEED=slow tests, or even more
>> extensive test suites like avocado-vt once in a while, but we
>> need to know who is running them, and when.
>>
>> Today, the only test set I know people really run and would
>> surely block a release is "make check [SPEED=quick]".
>>
>> So, for anybody that runs automated QEMU tests once in a while,
>> can we know:
>>
>> * What test cases are you running?  Where can we get more
>>   information about the tests you run?
>> * When do you run them?  What triggers a new test run?
> 
> I manually run qemu-iotests on release candidates:
> 
>  $ (cd tests/qemu-iotests && ./check && ./check -qcow2)
> 
> The goal is to identify test failures that still need to be addressed
> before the release is made.
> 
> I think Kevin Wolf and John Snow have also been running qemu-iotests to
> eliminate regressions during the freeze.
> 
> Stefan
> 

Yeah, it's always manual though.

I generally do an x86-for-x86 build with clang, make check, and do an
iotests across raw and qcow2 before sending a patch series.

When RC0 hits, I usually do a full build of all targets and do an
iotests check across all the file formats, but I don't bother with
protocols.

I generally don't use gcc because there's probably enough coverage out
there.

--js