bug#23205: 25.0.92; sort-tests is flaky

[Philipp Stephani]

On Sun, Apr 3, 2016 at 4:58 PM Eli Zaretskii <eliz@gnu.org> wrote:

> From: Philipp Stephani <phst@google.com>
> Date: Sun, 03 Apr 2016 10:30:10 +0000
>
> In the emacs-25 branch, in the directory test/automated, run:
>
> declare -i failures=0
> for ((i=0; i<1000; i++)); do
> if ! output="$(make sort-tests 2>&1)"; then
> ((failures++))
> echo "Iteration $i: $output"
> fi
> done
> echo "Failures: $failures"
>
> There will be typically a small number of failures (< 10), caused by
> e.g.
>
> Test sort-tests--lines condition:
> (ert-test-failed
> ((should
> (string=
> (substring ... 0 -1)
> sorted-words))
> :form
> (string= "Zi
> zI" "zI
> Zi")
> :value nil))
>
> This is presumably due to non-stable sorting. Either the sorting should
> be made stable, or the test shouldn't assume stable sorting.

Do you sometimes see such failures if you just run the test once?

I don't know, what's the difference between running it once and running it 1000 times? Since it fails randomly, it could fail in any iteration, including the first one. 

--

Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle

Diese E-Mail ist vertraulich. Wenn Sie nicht der richtige Adressat sind, leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen Sie die E-Mail und alle Anhänge. Vielen Dank.

This e-mail is confidential. If you are not the right addressee please do not forward it, please inform the sender, and please erase this e-mail including any attachments. Thanks.