Re: [Qemu-devel] [PATCH] Acceptance tests: host arch to target arch name mapping

[Eduardo Habkost]

On Wed, Oct 17, 2018 at 06:47:41PM -0400, Cleber Rosa wrote:
> 
> 
> On 10/17/18 6:15 PM, Eduardo Habkost wrote:
> > On Wed, Oct 17, 2018 at 04:59:25PM -0400, Cleber Rosa wrote:
> >> On 10/17/18 4:46 PM, Murilo Opsfelder Araujo wrote:
> > [...]
> >>> Then avocado could multiplex these variants file and call 
> >>> ./tests/acceptance/run
> >>> for each value of qemu_bin.  `run` script could skip and return if 
> >>> $qemu_bin
> >>> doesn't exist.  This approach also allows user forcing a value of 
> >>> qemu_bin when
> >>> calling `run` manually, for example:
> >>>
> >>>     ./tests/acceptance/run --qemu_bin=/path/to/your/qemu-system-blah ...
> >>>
> >>> This ./tests/acceptance/run can serve as an entry point to run all the 
> >>> tests.
> >>> If more parameters are considered mandatory in the future, the logic can 
> >>> be
> >>> placed there.
> >>>
> >>
> >> I'm completely favorable to having extra scripts or make rules that
> >> define a standard "job" behavior.  In fact, I think we'll end up having
> >> many of those.  "make check-acceptance" is just the first one.
> >>
> >> But being able to running individual tests should still be possible, and
> >> easy IMO.
> > 
> > Agreed.  But is there anything that requires "running an
> > individual test" to be synonymous to "running a test against only
> > one QEMU binary"?
> > 
> 
> I consider, generally speaking:
> 
>  * the test: the (parameterizable) code
>  * running an individual test: the execution of that code, once, with
> one parameter

This is where we're talking about completely different things.
When developers say "I want to run the test I have written", they
are probably expecting that code to run multiple times, each time
with different parameters.

If we implement this as multiple Avocado test cases, or in a
separate layer, it doesn't matter as long as it's easy and
obvious to run them.

> 
> There may be tests in which the logic of *one test* may require
> executing different QEMU binaries, but that is the exception, not the
> rule IMO.
> 
> Given that the acceptance tests are Avocado tests, I think it'd be a
> mistake (or very hard) to try to make "an individual (Avocado) test" use
> all the built QEMU binaries.  Variants (one for each unique set of
> parameters) is the natural thing to use here.

I trust your advice regarding the Avocado API.  If it's not good
practice, we don't need to make an individual Avocado test case
use built QEMU binaries.  We just need to provide an easy way for
developers to run the test they have written against all the
binaries.

If we can make that work with "avocado run mytest.py" it would
be nice, but not a must.

If we can make that work with "./tests/acceptance/mytest.py", it
would be great, but not a must.

If we need to make that work with "./tests/acceptance/run
mytest.py", it's good enough.


> 
> > We seem to have a terminology problem here: "I'm running one
> > individual test" may mean something to an user, but mean
> > something completely different to somebody thinking about Avocado
> > test cases.  Is this the source of our disagreement?
> > 
> 
> It could be, I'm certainly biased by the Avocado definition of tests and
> my limited understanding what others here may mean when they say "a
> test".  For instance, it may be that, to some, "make check" is
> considered "a test", and consequently, needs to run with all built QEMU
> binaries.

Probably "make check" wouldn't be called "a test", but:
* "tests/acceptance/mytest.py" is probably going to be called
  "a test".
* "I want to run mytest.py" would probably mean
  "I want to run mytst.py against all built QEMU binaries",
  not "I want to run one single Avocado test case with mytest.py".

-- 
Eduardo