Re: [Discuss-gnuradio] costas ambiguity and correlate-and-sync block in qpsk

[Landsman, Arik]

Hi Andy, 

Still looking it over, but one thing jumps out right away - for some reason I 
couldn't execute either Rx_syncd_2.grc or the Rx_syncd.py. getting the same 
error in both cases: 

""
Generating: '/home/ubuntu/Desktop/MSProject/alternateVersions/Rx_syncd_2.py'

Executing: '/usr/bin/python2 -u 
/home/ubuntu/Desktop/MSProject/alternateVersions/Rx_syncd_2.py'

Using Volk machine: sse4_1_64
Thrift: Sat Apr  2 19:05:54 2016 TServerSocket::listen() BIND 9090
terminate called after throwing an instance of 
'apache::thrift::transport::TTransportException'
  what():  Could not bind: Transport endpoint is not connected
""

does this look familiar by any chance?


Also could you elaborate a bit on what you did with 

""
        $ source <path_to_target>/setup_env.sh
        $ ./Rx_syncd.py | grep -v volk | sed -e 's/(//g' -e 's/)//g' > foo.txt
        $ octave
        octave:1> x = csvread('foo.txt');
        octave:2> t = [1:length(x)];
        octave:3> plot(t,abs(x))
        octave:4> plot(t,arg(x)/pi)
""

at the very least where can I find setup_env.sh?.. wondering what 
<path_to_target> may be. 
I suppose you then pipe a portion of the stream in Rx_syncd.py into foo.txt to 
parse with octave? 
going to try it if I find setup_env.sh on my end. and once I can run 
Rx_syncd.py... )

Thanks,
Arik 




________________________________________
From: Andy Walls address@hidden
Sent: Friday, April 01, 2016 9:18 AM
To: Landsman, Arik
Cc: address@hidden
Subject: Re: [Discuss-gnuradio] costas ambiguity and correlate-and-sync block 
in qpsk

Hi Arik,

I have attached a slightly improved flowgraph and a hand edited .py
file.

The only modifications to the flowgraph from before, is that I set the
reference samples for the corr_est block to start at the _middle_ of the
first symbol of the preamble and stop at the _middle_ of the last symbol
in the preamble; I also ensured the phase_est and time_est tags stayed
marked at the center of the last symbol of the preamble.  This has two
benefits:

1. The phase_est value will not be thrown off by the end of symbol
transition to a different unknown symbol after the preamble.
2. The correlation won't be sensitive to the unknown symbols before and
after the preamble.

In other words, it was a mistake to have the correlation reference
sequence start and end at the symbol transition times. :P

In the generated *.py script I added one line near the end:

    print tb.get_modulated_preamble_untrimmed()

So you can so this:

        $ source <path_to_target>/setup_env.sh
        $ ./Rx_syncd.py | grep -v volk | sed -e 's/(//g' -e 's/)//g' > foo.txt
        $ octave
        octave:1> x = csvread('foo.txt');
        octave:2> t = [1:length(x)];
        octave:3> plot(t,abs(x))
        octave:4> plot(t,arg(x)/pi)

and see why I picked the values I did to trim the modulated preamble to
the correct set of samples.

The modulator that generates the reference sequence has a pretty long
delay, and the first copy of the preamble it outputs is a little
distorted at the first symbol.  That's why I snipped out the second copy
of the preamble.

Regards,
Andy


On Thu, 2016-03-31 at 22:04 -0400, Andy Walls wrote:
> Hi Arik,
>
> Try the attached flowgraph.  I didn't get to put everything back in, but
> it demonstrates the principal.
>
> 1. Observe the corr_est block correlates to your preamble bytes.
> 2. Slide the phase offset around:
> a. observe the phase changing/wrapping in the time plot of the output of
> the correlator
> b. observe the phase_est tag value approximately matches the value of
> the phase offset slider.
> c. Observe the constellation rotate as you change the phase offset.
> d. Observe your decoded bits stay the same.
>
> Regards,
> Andy
>
>
> On Wed, 2016-03-30 at 16:56 -0400, Andy Walls wrote:
> > Hi Arik,
> >
> > I just took a quick look at your flowgraph.
> >
> > Some things jump out at me:
> >
> > 1. You have the modulator set at 2 samples/symbol, but the reality is,
> > most hardware is not going to transmit at 320 ksps.  The X310 USRP for
> > example requires a minimum of 500 ksps for Tx, so you might as well just
> > upsample and get more samples/symbol anyway.
> >
> > 2. You have complex channel models in your flowgraph, but there's not
> > much point in using them until the basic system with a perfect channel
> > works properly.  Disable them and save the headaches until you're ready.
> >
> > 3. Power Squelch followed by AGC is going to be a disaster.  AGC will be
> > at *maximum* gain during most of the squelch periods.  When your signal
> > shows up and breaks squelch, the AGC going to take time to come down
> > from that enormous gain.  Get rid of the squelch.  Squelch is for
> > humans, not modems.
> >
> > I'll look at it more later tonight.
> >
> > -Andy
> >
> > On Tue, 2016-03-29 at 23:48 +0000, Landsman, Arik wrote:
> > > Hi Andy,
> > >
> > > I like the "GMSK 9600" approach you mentioned - certainly will be looking 
> > > into it. in the meanwhile I attached my flow graph. The debug folder has 
> > > the input and output files.
> > >
> > > 0xFC0C3 is the sync word as can be seen in "stream_helloWorld". I use 
> > > Bless to generate the binaries but any hex editor would work I imagine.
> > >
> > > changing sps from =2 to =8 changes corr (as expected), so generating corr 
> > > symbols with a "matched" modulator seems like the way to go.
> > >
> > > changing threshold from 0.3 to say 0.5 forces a tag only on that first 
> > > ultra high spike (30k or so). but 0.3 has the same spike, just more tags 
> > > follow (which is encouraging).
> > >
> > > Tried bypassing the channel sim, power squelch, AGC, FLL directly into 
> > > the corr_est block in case the blocks change the symbols in an 
> > > unpredictable way - nothing seems to be off.
> > >
> > > As the ultimate check I usually loop the input file ("packet" if you 
> > > will) and compare it at the Rx. "Hello World" pops here and there but not 
> > > often enough to start considering BER testing for now.. costas output 
> > > flips by 90* sporadically.
> > >
> > > finally, the gui plots are not in the order of the signal chain but are 
> > > labeled properly. Hopefully easy to follow.
> > >
> > > And thanks again!
> > >
> > > Arik
> > >
> > >
> > > ________________________________________
> > > From: Andy Walls address@hidden
> > > Sent: Friday, March 25, 2016 6:45 PM
> > > To: Landsman, Arik
> > > Cc: address@hidden
> > > Subject: Re: [Discuss-gnuradio] costas ambiguity and correlate-and-sync 
> > > block in qpsk
> > >
> > > On Fri, 2016-03-25 at 22:14 +0000, Landsman, Arik wrote:
> > > > Hi Andy,
> > > >
> > > > I gave it a few more touches since we last spoke, but to no avail...
> > > > Costas still flips sporadically on a simulated channel.
> > > >
> > > > Generally the corr_est block would show a large spike in |corr|^2 on
> > > > the very first packet I send (>1600), but on all subsequent loops
> > > > (same packet) the value diminishes to ~100 with multiple peaks.
> > >
> > > That doesn't seem right.
> > >
> > >
> > > > I tried various preamble formats and sizes, including an 8 byte long
> > > > version to avoid any possible corr with payload. corr^2 certainly
> > > > changes but results in either multiple peaks at low levels (say 100
> > > > and 140), or with a huge spike at start as mentioned.
> > > >
> > > > I can generate  tags by reducing the threshold to extremely low values
> > > > (say 0.1, 0.3), which I suppose makes sense with the delta between the
> > > > first and subsequent spike levels.
> > > >
> > > > In either case there are more than one tag generated per packet as the
> > > > packet loops. Only one preamble per packet, non repeating.
> > > >
> > > >
> > > > matched filter taps - can the corr_est block accept taps in any way?
> > >
> > > No.
> > >
> > > The idea with corr_est is to give it exactly the reference samples that
> > > you are looking to match against.
> > >
> > > That usually means building them with one of the gnuradio modulator
> > > blocks *and* discarding the the initial samples which are a transient
> > > emitted during the initial delay of the gnuradio particular modulator.
> > >
> > > Look at the GRC file I just posted to the list regarding "GMSK 9600
> > > baud".  I had a modulator build the preamble samples. I had to add in a
> > > few flush/fill bits at the end of the preamble, which never get emitted
> > > by the modulator. I had to discard the initial transient emitted by the
> > > modulator.
> > >
> > > >  I'd like to hold sps=8 at least for the time being, and without a
> > > > filter there is little correlation between the stream and the
> > > > reference preamble. sps=2 was too low for timing recovery without a
> > > > preamble.  what do you generally use? (btw all of the debug above was
> > > > with sps=2).
> > >
> > > I use sps=10 with low data rate stuff.
> > >
> > > >  I am told I've been loosing sleep over this.. in any case I'll keep
> > > > trying but in the meanwhile if you had any comments I would deeply
> > > > appreciate it.
> > >
> > > If you have a flowgraph that you could share that simulates the input,
> > > I'd would probably be easier for me to see what's wrong about the
> > > corr_est process.
> > >
> > > Regards,
> > > Andy
> > >
> > > > Many Thanks,
> > > > Arik
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From: Landsman, Arik
> > > > Sent: Monday, March 14, 2016 11:59 AM
> > > > To: Andy Walls; address@hidden
> > > > Subject: RE: [Discuss-gnuradio] costas ambiguity and correlate-and-sync 
> > > > block in qpsk
> > > >
> > > > Hi Andy,
> > > >
> > > > Thank you for the detailed answer, I am certainly going to try what you 
> > > > suggest in the next couple of days. will post with updates as 
> > > > progressing along.
> > > >
> > > > As a side, I tried to hack the corr_and_sync block using a preamble 
> > > > which in BPSK looks the same as both I and Q of the QPSK data (e.g. 
> > > > [1,-1] in bpsk is [1+j,-1-j] in qpsk). This does not quite work in 
> > > > practice, I was getting sinusoids with very weak correlation (amplitude 
> > > > in 30's 40's) from the corr port. Strange, since the block should 
> > > > ignore the imaginary portion of the stream anyways since designed for 
> > > > BPSK.
> > > >
> > > > in terms of a clock pattern for preamble, looks like Barker codes is 
> > > > the way to go, at least statistically speaking for low correlation with 
> > > > any random "preamble-like" bit combinations in the payload. So avoiding 
> > > > clock patterns for sure.
> > > >
> > > > Will look to tag the end of preamble, from your comments it appears to 
> > > > be the more universal approach. In my case freq lock should be fairly 
> > > > good though,  channel is largely stationary, and I am using a FLL 
> > > > (although it does require longer bit combinations that are to result in 
> > > > a balanced spectrum to detect band edges). point well noted though.
> > > >
> > > > You mentioned M&M for timing recovery, have you tried the Polyphase  
> > > > block?  without passing tags, I had better results with the latter for 
> > > > whatever reason.  MM is more efficient though as I read.
> > > >
> > > > Just to clarify, were you capturing the time_est tags, or just relying 
> > > > on a combination of phase_est for costas and MM converging fast enough 
> > > > ahead of payload?
> > > >
> > > > finally, would you know of any good references for passing tags in 
> > > > gnuradio? I will certainly search around, but in case anything comes to 
> > > > mind (?)
> > > >
> > > > Best Regards,
> > > > Arik
> > > >
> > > >
> > > >
> > > > P.S. - In case it helps future causes - see "Phase-ambiguity resolution 
> > > > for QPSK modulation systems" by Nguyen, Tien Manh, NASA, 1989. Part1 
> > > > outlines the basic two approaches for correcting phase, which are diff 
> > > > encoding or preamble correlation. The preamble algorithm there is very 
> > > > much similar to the one Andy describes, although likely implemented as 
> > > > an IC (aka ASSP). In this case the author suggests diff encoding for 
> > > > burst transmissions, since bits are wasted on a preamble (which 
> > > > theoretically has 2x BER compared to diff encoding).
> > > >
> > > > That said, if phase rotation occurs in mid-payload the full packet is 
> > > > lost, unless some form of post-processing is used to recover - so at 
> > > > higher SNR diff encoding is possibly better. But I am yet to find a 
> > > > publication that compares real data on the topic.
> > > >
> > > > http://ntrs.nasa.gov/search.jsp?N=0&Ntk=All&Ntt=Phase-ambiguity%20resolution%20for%20QPSK%20modulation%20systems&Ntx=mode%20matchallpartial
> > > >
> > > >
> > > >
> > > >  ________________________________________
> > > > From: Andy Walls address@hidden
> > > > Sent: Sunday, March 13, 2016 8:51 AM
> > > > To: address@hidden
> > > > Cc: Landsman, Arik
> > > > Subject: Re: [Discuss-gnuradio] costas ambiguity and correlate-and-sync 
> > > > block in qpsk
> > > >
> > > > On Sun, 2016-03-13 at 01:35 -0500, address@hidden
> > > > wrote:
> > > > > Message: 1
> > > > > Date: Sat, 12 Mar 2016 21:34:03 +0000
> > > > > From: "Landsman, Arik"
> > > >
> > > > >
> > > > > Hello folks,
> > > > >
> > > > > I am trying to resolve the 90* ambiguity of costas for a QPSK
> > > > > receiver, and was hoping folks could weigh-in in case anyone had
> > > > > success with this in the past.
> > > > >
> > > > > yes, diff encoding works.. :) trying to make it work without though.
> > > > >
> > > > > Also, I had seen Tom R's example of his cor-and-sync block
> > > > > implementation in BPSK (but not qpsk). maybe the block could be
> > > > > "hacked" to support qpsk, such as by passing the preamble as bpsk but,
> > > > > say, upsampling the block to make the generated complex reference
> > > > > align with the incoming qpsk stream. going to try this when I get home
> > > > > tonight.
> > > > >
> > > > > Since Trx will be bursty and will use a preamble anyways, another
> > > > > thought was to correlate the stream with the 4 possible versions of
> > > > > the preamble (i.e. constellation rotations), and pass the best
> > > > > candidate downstream to select the proper constell object for demod
> > > > > (as opposed to adjusting costas, as in cor-and-sync block), or use the
> > > > > result for post-processing the incorrectly demodulated data. But both
> > > > > seem a bit indirect or wastefull..
> > > > >
> > > > > Any thoughts?
> > > >
> > > > Using a preamble makes sense to me.
> > > >
> > > > Do not use the correlate_and_sync block; it is unreliable and provides
> > > > and errant "phase_est" value.
> > > >
> > > > Use the corr_est block; it does correctly and more generically what the
> > > > correlate_and_sync block aimed to do.
> > > >
> > > > To use the corr_est block:
> > > >
> > > > 1. Generate a vector of samples which represents your preamble.  A
> > > > test_corr_est.grc file can be found here:
> > > >
> > > > https://github.com/gnuradio/gnuradio/tree/master/gr-digital/examples/demod
> > > >
> > > > which should give an example of how to use gnuradio modulator blocks to
> > > > build the preamble samples vector for you.  You could also use, pyhton,
> > > > Matlab, octave, or whatever.
> > > >
> > > >
> > > > 2. Tell the corr_est block where on the preamble you want it to place
> > > > the tags.  You must give the tag delay in units of samples.  Common
> > > > choices are:
> > > > a. start of preamble
> > > > b. middle of first symbol after start of preamble
> > > > c. middle of last symbol before end of preamble
> > > > d. end of preamble
> > > >
> > > > 3. The corr_est block outputs the following tags:
> > > >
> > > > corr_start: that start of your preamble
> > > >
> > > > corr_est: the absolute value of the correlation squared
> > > >
> > > > phase_est: the phase offset between the reference preamble and the
> > > > received preamble at the sample corresponding to the correlation peak
> > > > (which happens at the end of the preamble).
> > > >
> > > > time_est: the fractional sample delay from the sample that is at the
> > > > correlation peak to where the actual coorelation peak (which happens in
> > > > between samples)
> > > >
> > > >
> > > > 4. If you are well synchronized in frequency, you can just apply the
> > > > phase_est correction to the whole burst with a phase rotation by
> > > > multiplying the burst by exp(1.0j*-phase_est_value), IIRC.
> > > >
> > > > If you are not well synchronized in frequency, the phase_est value is
> > > > only sensible to apply at the samples at the end of the preamble.
> > > > But that's enough to resync a tracking loop.
> > > >
> > > >
> > > > 5. Modify your downstream blocks to look for the above tags, and reset
> > > > their state as appropriate, to acquire and maintain synchronization.
> > > >
> > > > 6. BTW, avoid a preamble that has a clock pattern (1010101010..).  It
> > > > causes duplicate correlation peaks (which you then have to inspect the
> > > > corr_est tag for the highest value), and the M&M clock recovery block
> > > > really drifts off such preambles badly for some reason.
> > > >
> > > > > Thank you in advance,
> > > > > Arik
> > > >
> > > >
> > > > Regards,
> > > > Andy
> > > >
> > >
> > >
> >
>