[Discuss-gnuradio] USRP2 stops after several-days-continuous-operation

[ILKYOUNG KWOUN]

Hi, folks,

I'm testing my application with USRP2. 
After I verify the function works fine logically, I started aging test by letting the program run over a week.
Although it varies, the program hangs eventually after running several days. (from 2~3 days to 8 days)

The followings are part of my python code

# In the top block,
    ~~~
    self.u = usrp2.source_32fc(self.interface, self.MAC_addr)
    s2v = gr.stream_to_vector(gr.sizeof_gr_complex, self.fft_size)
    fft = gr.fft_vcc(self.fft_size, True, mywindow)
    c2mag = gr.complex_to_mag_squared(self.fft_size)
    self.msgq = gr.msg_queue(1)
    stats = gr.message_sink(gr.sizeof_float * self.fft_size, self.msgq, True)

    self.connect(self.u, s2v, fft, c2mag, stats)
    ~~~

# In main_loop thread
    ~~~
    while(1):
        rcvd_param = rx_msg.check_udp()

        if (rcvd_param[0] == 1):        # Valid UDP packet received
            tb.u.set_decim(cp.decim)    # Set(Update) USRP2 decim
            cp.init_ctrl_param(rcvd_param, tb.fft_size)

        radio.tune(cp.ctr_freq)                                # This is to control other independent receiver
        cp.spectrum_inv = radio.spectrum_status()

        # Force buffer flushing
        samples = msg.payload(tb.msgq.delete_head(), tb.fft_size)
        samples = msg.payload(tb.msgq.delete_head(), tb.fft_size)
        samples = msg.payload(tb.msgq.delete_head(), tb.fft_size)
        samples.make_rawdata()
        tx_msg.send_packet_over_UDP(cp, samples.raw_data)
    ~~~
        cp.update()                                                # End of while loop
    ~~~

In the program, I generated a message queue to get the sample from USRP2, do some process it a bit, and send it to other server over UDP.

In the while loop,
1. Check the user defined control parameter (concerning tuning frequency and USRP2 decim. rate)
    If any control message is delivered, it update the control parameters. Otherwise, skips.
2. Tune the independent receiver module. This is a frontend of USRP2.
3. Flushing sample queue
4. Get the sample out of queue and send it over UDP

The program hangs between first and second buffer flushing after several days of continuous operation.
When it hangs, I figured out that the USRP2 does not send sample data to PC. (I confirmed it with wireshark.)

I attached the program to gdb and found out that the program is stuck waiting for the queue to be filled, as you see in the following gdb back-trace message.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
(gdb) info program
        Using the running image of attached LWP 28651.
Program stopped at 0xa08a32e9.
(gdb) bt
#0  0x00007f26a08a32e9 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/libpthread.so.0
#1  0x00007f269f3ec9c0 in omni_condition::wait (this=<value optimized out>) at posix.cc:152
#2  0x00007f269f794c98 in gr_msg_queue::delete_head (this=0x2aea520) at gr_msg_queue.cc:81
#3  0x00007f269fa3b125 in gr_py_msg_queue__delete_head (q= {px = 0x7fffb561f150, pn = {pi_ = 0xffffffffffffffff}}) at gnuradio_swig_py_runtime.cc:4840
#4  0x00007f269fa43649 in _wrap_gr_py_msg_queue__delete_head (args=<value optimized out>, kwargs=<value optimized out>) at gnuradio_swig_py_runtime.cc:13477
#5  0x000000000041d3bd in PyObject_Call ()
#6  0x00000000004a1460 in PyEval_EvalFrameEx ()
#7  0x00000000004a4649 in PyEval_EvalCodeEx ()
#8  0x00000000004a2cb0 in PyEval_EvalFrameEx ()
#9  0x00000000004a3dae in PyEval_EvalFrameEx ()
#10 0x00000000004a4649 in PyEval_EvalCodeEx ()
#11 0x00000000004a4762 in PyEval_EvalCode ()
#12 0x00000000004c4c3c in PyRun_FileExFlags ()
#13 0x00000000004c4f6b in PyRun_SimpleFileExFlags ()
#14 0x00000000004189ce in Py_Main ()
#15 0x00007f269fca05a6 in __libc_start_main () from /lib/libc.so.6
#16 0x0000000000417ae9 in _start ()
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

So, I am suspicious if this is a problem caused simply by the USRP2 not sending data.
Is there anybody experienced similar case or having any idea?

Regards,

ILKYOUNG.