Re: [Discuss-gnuradio] make test failure at qa_constellation_receiver

[Ben Reynwar]

It appears that each test case is being run twice.  I think once to
generate the xml output, and once to generate output for stdout and
tell you whether it failed.  The random number generator isn't being
reseeded at the start of each test so they can produce different
results, so you can sometimes see debug statements that indicate it
should fail, but the output to stdout claims that is passes.  I've
attached a patch that make each test use it's own random number
generating object which seems a tidier way to do this, and makes the
tests repeatable.

Since there seem to be a fair number of choices of seed that produce
an error fraction of over 0.2, I've also increased the acceptable
error rate in the test to 0.3, as Alick initially suggested.

On Sat, Apr 14, 2012 at 9:39 AM, Alick Zhao <address@hidden> wrote:
> On Sat, 14 Apr 2012 13:40:35 +0200, Martin Braun wrote:
>> On Sat, Apr 14, 2012 at 05:20:13PM +0800, Alick Zhao wrote:
>>> Yes the bug still occurs.
>>>
>>> I just wrote a simple script to ran the test for 50 times on T60 with
>>> Ubuntu, GNU Radio 3.5.3 equipped, before and after FREQUENCY_OFFSET set
>>> to 0. All failed the test. (n_pass is 0/50 in both sets) Every test's
>>> output is almost identical in each set except particular test time
>>> length. The ones before FREQUENCY_OFFSET change is basically the same as
>>> test.t60.log already attached. One of the ones with FREQUENCY_OFFSET set
>>> to 0 is attached below. I guess the almost same contents of output is
>>> due to fixed random seed.
>>>
>>> I also ran the script on a Dell desktop with Fedora, and the result is
>>> 50/50 pass the test. Then I notice one line in the qa python file says
>>> that seed 1234 fails. However, 50/50 are OK with seed 1234 on the Dell
>>> desktop.
>>
>> So,
>>
>> I tried this on a native 32-Bit Ubuntu 11.10, and it fails (tells me
>> it's using "Volk machine: sse3_32".
>>
>> I also noticed the line that says seed=1234 fails. Also, the seed is set
>> multiple times in the script. If this is the source of the bug or not,
>> it should be fixed, because the seed is reset somewhere in the guts of
>> the test. I'll post a patch on patch-gnuradio--this eliminates the
>> problem on my machine, for some reason. If it does the same for you,
>> this might actually be the solution.
>>
>> MB
>>
>>
>
> I applied the patch and tested it. No failure, yeah!
>
> However, with my debugging line, I can see this:
>
> [...]
> constellation: <constellation psk (m=32)>  differential: True  correct:
> 0.942307692308
> constellation: <constellation psk (m=64)>  differential: True  correct:
> 0.772435897436
> constellation: <constellation psk (m=2)>  differential: True  correct: 1.0
> [...]
>
> So why 0.77XXX on the second line does not cause the assert to fail?
>
> alick
>
> _______________________________________________
> Discuss-gnuradio mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

0001-Improving-qa_constellation_receiver.patch
Description: Binary data