Re: [PATCH v2 19/22] tests/qtest/migration: Add migration-test-smoke

[Peter Xu]

On Fri, Dec 20, 2024 at 03:34:32PM +0000, Daniel P. Berrangé wrote:
> On Fri, Dec 20, 2024 at 10:18:37AM -0500, Peter Xu wrote:
> > On Thu, Dec 19, 2024 at 04:31:04PM -0300, Fabiano Rosas wrote:
> > > We shouldn't change stuff that's also used by the rest of the
> > > community. People know about QEMU_TEST_FLAKY_TESTS and -m slow. These
> > > must continue to work the same.
> > 
> > I see what I overlook; it's used much more than I thought in qtest and we
> > also have a CI for it.. So ok, let's keep at least QEMU_TEST_FLAKY_TESTS.
> > 
> > But again, I don't think it matters much even if we rename it, it means the
> > flaky CI test won't run these two migration tests, but that's not the end
> > of the world either, if you see what I meant.  CI relies on the normal
> > tests rather than flaky tests to present.
> > 
> > We should be able to move in / take out FLAKY tests at will, as that's not
> > what CI is really relying on.  Here renaming the macro in migration test
> > almost means we take both ou.t
> 
> Side-note - QEMU_TEST_FLAKY_TESTS is something we should apply
> consistently across all types of tests - unit, qtest, functional,
> and across all environments - CI and local developer execution.
> 
> In recent changes to functional testing, I've set the expectation[1]
> that any use of QEMU_TEST_FLAKY_TESTS *must* be accompanied by a
> link to the gitlab.com/qemu/qemu-project  issue that describes
> the flaky behaviour seen. We've got too many places with flaky
> tests where we don't quiet remember what was flaky, so don't
> know if we can remove it or not

Thanks for the heads up.  Yes it makes sense to document it explicitly in
the code.  For now we can still dig into git log, but not as easy as
inline.

Aside that, IIUC the major challenge will still be there though on the
justification itself.  It could depend on how easily reproduceable the
issue is on the developers' hosts.. If there can be some link that not only
report where does it came from, but also to verify a fix that'll be even
nicer.

-- 
Peter Xu