Re: [Discuss-gnuradio] make test failure at qa_constellation_receiver

[HANDLER_OUTPUT]
Test project /home/niulab/gnuradio/build3.5.3

Start 82: qa_constellation_receiver
1/1 Test #82: qa_constellation_receiver ........***Failed 3.41 sec
>>> gr_fir_ccf: using SSE
>>> gr_fir_ccc: using SSE
constellation: <constellation psk (m=2)> differential: True correct: 1.0
constellation: <constellation psk (m=2)> differential: True correct: 1.0
constellation: <constellation psk (m=4)> differential: True correct: 1.0
constellation: <constellation psk (m=4)> differential: True correct: 1.0
constellation: <constellation psk (m=8)> differential: True correct: 0.982300884956
constellation: <constellation psk (m=8)> differential: True correct: 1.0
constellation: <constellation psk (m=16)> differential: True correct: 0.96837944664
constellation: <constellation psk (m=16)> differential: True correct: 0.990118577075
constellation: <constellation psk (m=32)> differential: True correct: 0.957746478873
constellation: <constellation psk (m=32)> differential: True correct: 0.93896713615
constellation: <constellation psk (m=64)> differential: True correct: 0.821596244131
constellation: <constellation psk (m=64)> differential: True correct: 0.863849765258
constellation: <constellation bpsk> differential: True correct: 1.0
constellation: <constellation dqpsk> differential: True correct: 1.0
constellation: <constellation psk (m=2)> differential: True correct: 1.0
constellation: <constellation psk (m=2)> differential: True correct: 1.0
constellation: <constellation psk (m=4)> differential: True correct: 1.0
constellation: <constellation psk (m=4)> differential: True correct: 1.0
constellation: <constellation psk (m=8)> differential: True correct: 0.977876106195
constellation: <constellation psk (m=8)> differential: True correct: 0.893805309735
constellation: <constellation psk (m=16)> differential: True correct: 0.990118577075
constellation: <constellation psk (m=16)> differential: True correct: 0.990118577075
constellation: <constellation psk (m=32)> differential: True correct: 0.951486697966
constellation: <constellation psk (m=32)> differential: True correct: 0.981220657277
constellation: <constellation psk (m=64)> differential: True correct: 0.801251956182
constellation: <constellation psk (m=64)> differential: True correct: 0.735524256651
F
======================================================================
FAIL: test_basic (__main__.test_constellation_receiver)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/niulab/gnuradio/gr-digital/python/qa_constellation_receiver.py", line 101, in test_basic
self.assertTrue(correct > REQ_CORRECT)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1 test in 1.438s

FAILED (failures=1)
Using Volk machine: ssse3_32

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 3.43 sec

The following tests FAILED:
82 - qa_constellation_receiver (Failed)
[ERROR_MESSAGE]
Errors while running CTest

#!/bin/bash

prefix=${1:-log}
n_ok=0
for cnt in $(seq 50); do
if ctest -Q --output-on-failure -R qa_constellation_receiver \
-O ${prefix}/ctest.log.$cnt; then
let ++n_ok
fi
done
echo "right/total: $n_ok/50"
On Fri, 13 Apr 2012 13:20:01 -0700, Ben Reynwar wrote:
> 2012/4/12 Alick Zhao <alick9188@gmail.com>:
>> Hi all,
>>
>> Recently I have upgraded my gnuradio build to v3.5.3 on several
>> computers, and I find that on two machines with Ubuntu 11.10, make test
>> will fail the test qa_constellation_receiver while on the other two with
>> Fedora 16 all tests are passed.
>>
>> To investigate the problem, I add one line in the file
>> `gr-digital/python/qa_constellation_receiver.py` which just print the
>> value of constellation, differential, and correct before the assert.
>> Then I run the script
>> `build/gr-digital/python/qa_constellation_receiver_test.sh`. The output
>> is recorded and attached below. Among them:
>>
>> * test.t41.log is from a Thinkpad T41 with Ubuntu 11.10 32bit installed,
>> * test.t60.log is from a Thinkpad T60 with Ubuntu 11.10 32bit installed,
>> * test.f16.log is from a HP 6531s with Fedora 16 x86_64 installed.
>>
>> Hope these logs are helpful to diagnose the problem.
>>
>> PS: As a side note, my debugging line's output appears later than the
>> result, which is not the case for screen output. I think this is
>> probably related to stdout buffering.
>>
>> PS2: I once changed the REQ_CORRECT to 0.7 on one Ubuntu machine,
>> and then made the test passed. I wonder if it is a valid fix.
>>
>
> It sounds like this is definitely a bug, but it's hard for me to track
> it down because I can't replicate it. Does the bug still occur if you
> set FREQUENCY_OFFSET = 0 in the test case?

Yes the bug still occurs.

I just wrote a simple script to ran the test for 50 times on T60 with
Ubuntu, GNU Radio 3.5.3 equipped, before and after FREQUENCY_OFFSET set
to 0. All failed the test. (n_pass is 0/50 in both sets) Every test's
output is almost identical in each set except particular test time
length. The ones before FREQUENCY_OFFSET change is basically the same as
test.t60.log already attached. One of the ones with FREQUENCY_OFFSET set
to 0 is attached below. I guess the almost same contents of output is
due to fixed random seed.

I also ran the script on a Dell desktop with Fedora, and the result is
50/50 pass the test. Then I notice one line in the qa python file says
that seed 1234 fails. However, 50/50 are OK with seed 1234 on the Dell
desktop.

alick