Re: [Discuss-gnuradio] Correlation Estimator Over the Air

[Andy Walls]

On Tue, 2015-10-20 at 12:48 -0700, Richard Bell wrote:
> Andy, Sean,
> 
> 
> Even if you use a sequence that has good correlation properties, you
> will sometimes get a double tag. This happens when the correlation
> threshold is passed just at the end of a call to work. Upon the next
> call to work, the very first few samples may still be above the
> threshold, but the correlator has no way of knowing this was actually
> due to the peak it already tagged in the previous call to work, so it
> tags it again. 
> 
> 
> Andy, how do you work around double or triple tags usually? The tag
> location in all my applications is vitally important to the rest of
> the system. I couldn't handle ambiguity in that location. Is your
> system just not as dependant on knowing exact frame start positions,
> or are there other helpers to find this exactly down the line in your
> systems? Very curious.

Both not as dependent and other helpers. :)

For AIS GMSK timing recovery for example, I just need to know the center
of a symbol to reset the timing recovery loops downstream.  Any one of
the up to 4 correlations is good enough, so usually I use the first one
in the group.

For AIS TOA, I usually skip the 1st corr_est tag and tag the second one
with the precise TOA of that correlation.  Later, downstream after clock
recovery,  I can find the AIS start byte, figure out what bit of the
preamble that TOA tag is on, and then back up how every many bits the
real start of the preamble, and adjust the TOA time by that many bit
times. 

The point is, if you have blocks that can't handle multiple correlations
clustered on a preamble, then write at least one downstream block that
will filter out all of the ones in the cluster you don't like.

So how did I decide on the logic for which ones I liked?  As we say
around my office: "The first thing you do is plot the data."  

For determining parameters to minimize instances of multiple
correlations on a preamble, I just did things empirically.  IIRC, I set
my AGC to a ref level of 1.0-ish, looked at the plots of the abs("corr")
output, and picked a threshold number that would always give me at least
1 correlation for an AIS burst, while minimizing the number of multiple
correlations on a preamble.

FWIW, if the end of work() concerns you, in my original implementation,
I had a corr_eow tag emitted on the "corr" output.  In the final
version, it got commented out:
https://github.com/gnuradio/gnuradio/blob/master/gr-digital/lib/corr_est_cc_impl.cc#L323

Regards,
Andy

> 
> Rich
> 
> 
> 
> 
> On Tue, Oct 20, 2015 at 12:32 PM, Nowlan, Sean
> <address@hidden> wrote:
>         Logan, I've been away from email for the last week so sorry
>         for the delay.
>         
>         > Is the k value the reference value in the AGC2 block or the
>         max gain? Would you mind explaining the equation you used a
>         little more?
>         
>         Yes, it's the reference value. I think the max gain is used to
>         keep the AGC from blowing up.
>         
>         The equation I mentioned is essentially the Euclidean norm,
>         which we applied to our 4x oversampled, root-raised cosine
>         filtered, BPSK sequence of 0s and 1s mapped to constellation
>         points -1+0j and +1+0j, respectively. The corr_est block is
>         running at 4x oversampling of the baud rate. So for BPSK, you
>         can use either GNU Radio or Matlab/Octave to create a
>         root-raised cosine matched filter, map your preamble sequence
>         to constellation points, and filter them. Then calculate the
>         Euclidean norm of the upsampled, matched filtered sequence.
>         That's what I called "K" in my previous message. Dividing by
>         the length of this sequence gives you the average
>         "energy" (unitless) of the sequence. Think of it this way: the
>         corr_est block is a matched filter that will give maximum
>         likelihood detection when the input sample stream containing a
>         preamble sequence matches up in time and level with the
>         correlation sequence. So the AGC should be configured to
>         normalize the input stream to allow that to happen.
>         
>         Sean
>         ________________________________________
>         From: discuss-gnuradio-bounces
>         address@hidden <discuss-gnuradio-bounces
>         address@hidden> on behalf of Andy Walls
>         <address@hidden>
>         Sent: Tuesday, October 20, 2015 12:38 PM
>         To: address@hidden
>         Subject: Re: [Discuss-gnuradio] Correlation Estimator Over the
>         Air
>         
>         On Tue, 2015-10-20 at 11:46 -0400,
>         address@hidden
>         wrote:
>         > Message: 10
>         > Date: Tue, 20 Oct 2015 10:46:11 -0500
>         > From: "Washbourne, Logan"
>         > To: GNURadio Discussion List <address@hidden>
>         > Subject: Re: [Discuss-gnuradio] Correlation Estimator Over
>         the Air
>         
>         > Hello,
>         >
>         > I have still been working on getting the over the air
>         communication to
>         > work
>         > and I think I have definitely narrowed it down to the AGC
>         being the
>         > problem. I took Sean's advice and calculated the average
>         energy per
>         > sample
>         > for the preamble, which yielded me a value of .0078 Joules
>         and I used
>         > that
>         > for the reference value in the AGC2 block. When I used that
>         value as
>         > the
>         > reference, the correlation estimator wouldn't catch anything
>         as
>         > correlated.
>         > It is a real possibility that I calculated the energy per
>         sample
>         > incorrectly. I calculated it by only sending the preamble
>         over the
>         > air,
>         > then took the first 64 complex values( I used some code from
>         the
>         > octave
>         > files in gr-utils, to convert file sink data to complex
>         values) from
>         > the
>         > USRP Source block. I then found the magnitude of those
>         values, squared
>         > them, summed them, and then divided by 64. Dividing by 64
>         might be
>         > incorrect.
>         >
>         > That is what I tried in terms of calculating the reference
>         value of
>         > the
>         > AGC, but I also just plugged numbers into the block as well,
>         and what
>         > I
>         > seem to get most often is the correlation estimator block
>         spitting out
>         > several "correlated values" that are within a few samples of
>         each
>         > other,
>         > which ideally shouldn't happen seeing as how the payload
>         length is 992
>         > samples long and there is a null source injected roughly 32k
>         zeros
>         > into the
>         > communication stream.
>         
>         Well, it can happen, depending on your preamble.
>         
>         AIS has a preamble that is essentially:
>         110011001100110011001100111
>         I either get 4 or 2 hits on that every time, simply because
>         the preamble
>         is periodic and slightly shifting things still correlates well
>         enough.
>         
>         DSC is worse: 1010101010101.
>         
>         >
>         > I have some pictures of what my output graphs look like, I
>         am
>         > concerned
>         > that the correlation tags are being associated with the very
>         low
>         > amplitude
>         > values on the correlation plot.
>         
>         Try and zoom in on where the correaltions are happening and
>         see how many
>         symbols they are apart.
>         
>         Also plot a scaled version of the magnitude or magnitude^2 of
>         the "corr"
>         output delayed by the preamble length (in samples).  That will
>         let you
>         eyeball if multiple correlations can be expected.
>         
>         See the attached "clock pattern" preamble correlation.  The
>         premable is
>         101010101.. and the delayed, absolute value of the correlation
>         is
>         plotted above the baseband.  I get 2 peaks or 0 peaks no
>         matter where I
>         set the threshold -- no way around it based on the correlation
>         properties of the preamble.  (The plot might lead you to
>         believe I
>         should get 4 or 5 peaks, but the abs(corr) is flipping the
>         negative
>         correlation troughs to look like positive correlation peaks.)
>         
>         I perhaps could make the preamble longer and include some of
>         the start
>         byte bits, but then I would have to be sure of the bit
>         polarity of the
>         start byte bits.
>         
>         -Andy
>         
>         
>         
>         > These pictures are of the same communication session,
>         screen-printed
>         > roughly 5 secs apart.
>         >
>         > (If the attachments don't make it through the mailing list,
>         please let
>         > me
>         > know and I can host them elsewhere)
>         >
>         > I really appreciate your guys' help.
>         >
>         > Logan Washbourne
>         > Electrical Engineering Graduate Student
>         > (Electromagnetics)
>         >
>         >
>         > On Wed, Oct 14, 2015 at 1:31 PM, Washbourne, Logan <
>         > address@hidden> wrote:
>         >
>         > > Sean,
>         > >
>         > > Thanks for the tip! Is the k value the reference value in
>         the AGC2
>         > block
>         > > or the max gain? Would you mind explaining the equation
>         you used a
>         > little
>         > > more? From what I understand of it, the seq is the 1's and
>         0's that
>         > make up
>         > > the pseudorandom preamble, N is the sequence length and i
>         is the
>         > specific
>         > > value of the sequence when summing. Is that right or am I
>         way off
>         > base?
>         > >
>         > >
>         > >
>         > > Logan Washbourne
>         > > Electrical Engineering Graduate Student
>         > > (Electromagnetics)
>         > >
>         > >
>         > > On Wed, Oct 14, 2015 at 9:43 AM, Nowlan, Sean
>         > <address@hidden
>         > > > wrote:
>         > >
>         > >> ?We saw a lot of improvement in the performance of the
>         Correlation
>         > >> Estimator block once we calculated a level for the AGC.
>         In our
>         > case, we're
>         > >> looking for a BPSK preamble that is a pseudorandom
>         sequence. The
>         > corr_est
>         > >> block is provided with the match-filtered version of the
>         preamble
>         > sequence.
>         > >> So we calculated the average energy per sample of that
>         sequence as
>         > K =
>         > >> \frac{ \sum_{i}^{N}(\abs{seq}^2) }{ N }. (Sorry if the
>         LaTeX
>         > notation is a
>         > >> bit off). So K is the target level we use in the AGC2
>         block. This
>         > seems to
>         > >> work pretty well for us.
>         > >>
>         > >>
>         > >> Sean
>         > >>
>         > >>
>         > >> ------------------------------
>         > >> *From:* discuss-gnuradio-bounces
>         > address@hidden
>         > >> <discuss-gnuradio-bounces
>         address@hidden> on
>         > behalf
>         > >> of Washbourne, Logan <address@hidden>
>         > >> *Sent:* Tuesday, October 13, 2015 12:03 PM
>         > >> *To:* GNURadio Discussion List
>         > >> *Subject:* Re: [Discuss-gnuradio] Correlation Estimator
>         Over the
>         > Air
>         > >>
>         > >> Rich,
>         > >>
>         > >> Ah, that makes so much sense now. I was modulating all
>         that came
>         > out of
>         > >> the costas loop and packing the bits into bytes which
>         essentially
>         > means I
>         > >> was undoing the syncing the blocks before it were doing.
>         This
>         > morning I
>         > >> tried searching for the preamble in the binary stream
>         that was
>         > output from
>         > >> the constellation decoder and I found it. It was 98
>         samples in,
>         > which
>         > >> confused me, but now that I know I am looking for a tag
>         then that
>         > makes
>         > >> perfect sense.
>         > >>
>         > >> I just tried that same process, looking for the preamble
>         in the
>         > binary
>         > >> stream(using matlab) for the over the air program and I
>         still can't
>         > find
>         > >> it, but I am going to play around with the AGC block and
>         see if I
>         > can't
>         > >> tweak it enough to work.
>         > >>
>         > >> I really appreciate your help, I finally feel like I'm
>         getting
>         > close to
>         > >> my goal.
>         > >>
>         > >> Logan Washbourne
>         > >> Electrical Engineering Graduate Student
>         > >> (Electromagnetics)
>         > >>
>         > >>
>         > >> On Tue, Oct 13, 2015 at 10:55 AM, Richard Bell
>         > <address@hidden>
>         > >> wrote:
>         > >>
>         > >>> Logan,
>         > >>>
>         > >>> How are you doing this test to see if the bit stream
>         comes out?
>         > >>>
>         > >>> The corr_est block correlates against a known sequence
>         and when
>         > the
>         > >>> correlation output is above a threshold, tags the local
>         maximum
>         > item around
>         > >>> that beak. Now it's up to you to do something with that
>         tag. It
>         > places the
>         > >>> tag at the very front of the correlation sequence, so
>         one of the
>         > first
>         > >>> things you need to do is handle the tag placement. You
>         have some
>         > control
>         > >>> over that with the builtin delay paramter, but sometimes
>         that's
>         > not enough.
>         > >>>
>         > >>> I would recommend taking a step back and proving you
>         know how to
>         > do the
>         > >>> transformations that don't involve synchronization and
>         channel
>         > correction.
>         > >>> Take your input data, map it to symbols, modulate it and
>         then
>         > immediately
>         > >>> reverse the process. No channel, no noise, no
>         synchronization. If
>         > you can't
>         > >>> make this simulation work, you are misunderstanding
>         something.
>         > >>>
>         > >>> Rich
>         > >>>
>         > >>> On Mon, Oct 12, 2015 at 1:28 PM, Washbourne, Logan <
>         > >>> address@hidden> wrote:
>         > >>>
>         > >>>> Rich and others,
>         > >>>>
>         > >>>> I added the AGC block to RX side and after playing with
>         the
>         > parameters
>         > >>>> for awhile I got a correlation spike! My next step was
>         to confirm
>         > that my
>         > >>>> output equalled my input (byte-wise). In order to
>         accomplish
>         > this, I added
>         > >>>> a Constellation Decoder block after the costas loop and
>         used the
>         > >>>> constellation object as the input parameter. Then I
>         repacked the
>         > bits into
>         > >>>> 8 bit bytes and saved it to a file. I also saved the
>         input byte
>         > stream to a
>         > >>>> file. I looked at those files in Matlab and so far I
>         have not
>         > been able to
>         > >>>> find the preamble in the output byte stream.
>         > >>>>
>         > >>>> After not being able to determine if this communication
>         system
>         > was
>         > >>>> working( the TX and RX programs utilizing the USRPs), I
>         took a
>         > step back
>         > >>>> and tried to figure out how to confirm if the
>         test_corr_est.grc
>         > had the
>         > >>>> same output as its input and I'm running into the same
>         problem, I
>         > haven't
>         > >>>> been able to find the preamble in the output.
>         > >>>>
>         > >>>> Both programs show a clear correlation spike, I just
>         want to put
>         > my
>         > >>>> mind at ease and verify if it's working through the
>         actual bytes.
>         > One last
>         > >>>> note, the packed output byte stream is roughly 5.5
>         times smaller
>         > than the
>         > >>>> input byte stream, which I was expecting a really close
>         1:1 size,
>         > this
>         > >>>> makes me think I am overlooking a consequence of one of
>         the
>         > blocks, namely
>         > >>>> the Correlation Estimator, Polyphase Clock Sync, or the
>         Costas
>         > Loop.
>         > >>>>
>         > >>>> Does anyone have any suggestions?
>         > >>>>
>         > >>>> I know this thread is getting a little long, but I
>         appreciate
>         > >>>> everyone's patience with my questions.
>         > >>>>
>         > >>>>
>         > >>>>
>         > >>>> Logan Washbourne
>         > >>>> Electrical Engineering Graduate Student
>         > >>>> (Electromagnetics)
>         > >>>>
>         > >>>>
>         > >>>> On Wed, Oct 7, 2015 at 11:26 AM, Richard Bell
>         > <address@hidden>
>         > >>>> wrote:
>         > >>>>
>         > >>>>> Ah, yes. I suspect you don't have an AGC in your
>         flowgraph?
>         > Whenever
>         > >>>>> you're thresholding against some static number, you
>         need to be
>         > sure your
>         > >>>>> input signal is set to a known amplitude, which is
>         what the AGC
>         > does for
>         > >>>>> you. If you put an AGC in the chain you should see
>         peak values
>         > that get
>         > >>>>> close to your simulation values. The AGC produces a
>         stable
>         > platform for the
>         > >>>>> rest of your blocks to sit on.
>         > >>>>>
>         > >>>>> The AGC parameters can really play havoc with your
>         system. The
>         > AGC can
>         > >>>>> be the cause of a lot of headache. If you find
>         yourself
>         > debugging something
>         > >>>>> that makes no sense, often it's the AGC's fault, in my
>         > experience. I
>         > >>>>> recommend you create a simulation that stresses the
>         AGC and
>         > prove it will
>         > >>>>> work as best you can before going to over the air.
>         > >>>>>
>         > >>>>> Rich
>         > >>>>>
>         
>         
>         _______________________________________________
>         Discuss-gnuradio mailing list
>         address@hidden
>         https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>         
> 
>