Re: [PATCH 3/3] iotests: distinguish 'skipped' and 'not run' states

[Denis V. Lunev]

On 9/14/23 16:10, Eric Blake wrote:
> On Wed, Sep 13, 2023 at 10:36:35AM -0500, Eric Blake wrote:
> > On Wed, Sep 06, 2023 at 04:09:17PM +0200, Denis V. Lunev wrote:
> > > Each particular testcase could skipped intentionally and accidentally.
> > > For example the test is not designed for a particular image format or
> > > is not run due to the missed library.
> > >
> > > The latter case is unwanted in reality. Though the discussion has
> > > revealed that failing the test in such a case would be bad. Thus the
> > > patch tries to do different thing. It adds additional status for
> > > the test case - 'skipped' and bound intentinal cases to that state.
> > intentional
> >
> > > Signed-off-by: Denis V. Lunev <den@openvz.org>
> > > CC: Kevin Wolf <kwolf@redhat.com>
> > > CC: Hanna Reitz <hreitz@redhat.com>
> > > CC: Eric Blake <eblake@redhat.com>
> > > CC: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> > > ---
> > >   tests/qemu-iotests/common.rc     | 23 ++++++++++++++++-------
> > >   tests/qemu-iotests/iotests.py    | 19 +++++++++++++++----
> > >   tests/qemu-iotests/testrunner.py | 17 +++++++++++++++--
> > >   3 files changed, 46 insertions(+), 13 deletions(-)
> > I agree that this makes it easier to identify skipped tests that can
> > never be run (not designed for that combination) vs tests that can
> > plausibly be run by addressing missing dependencies (such as
> > installing additional libraries, a different file system, ...).
> >
> > Reviewed-by: Eric Blake <eblake@redhat.com>
> >
> > Because patch 1 impacts NBD testing, I can queue the series through my
> > NBD tree if no one else picks it up first.
> Given further conversation, I've removed patch 3 from my queue while
> conversation continues, but patch 1 and 2 are still queued (they are
> independently useful, whether or not we add another result category)
clear...

For me such a filter was very useful, as we
* have had problems in the past with the missed test cases due to 
non-satisfied reqs
* this separation gives positive feedback with test fixes (2 patches in 
the serie)
* there is no idea how this could be done in the ideal case and there is 
no idea
   how unclear ideal could be reached. Having something simple/useful looks
   preferred to me.

Den