Re: [Discuss-gnuradio] fast parallel filtering

[Dirk Gorissen]

Hi Andy, Marcus,

Thanks very much for taking the time to think about this.

Just to answer your questions:

>Dirk's problem was the low SNR in his recording, so he'd like to take
>the shape of his pulse into consideration, but he doesn't know the
>spectral position of the same – Dirk, can you confirm?

Yes, Im dealing with a very weak signal against quite a bit of
background noise so the more prior knowledge I can leverage the
better.

>Dirk, thinking about this, the relation of the spectral uncertainty to
>the bandwidth of the pulse would be interesting  – can you refresh our
>memory? I think the pulse was 10ms long (so, pretty long) but I forgot
>how it looked like.

Yes, 10ms long (this is pretty exact), occurring every 1.5 seconds, in
the 150MHz range. My input stream is coming from an SDR.
As an aside I actually have multiple transmitters each pulsing at
slightly different frequencies (e.g., 150.10, 150.22, ...) but Im
happy to treat them independently so you can ignore that and assume
there is just one.

You can see pictures of the pulse (taken a while back, for a specific
tx frequency) here: https://goo.gl/photos/Y3Ea6kJo1ewrcDYM8

Note though that this is taken with the transmitter very close to the
antenna so the signal is unrealistically strong. So its just to
illustrate.

I also quickly captured a file with raw IQ samples (File sink) in case
anybody is interested. It starts with the transmitter very close to
antenna, moves progressively further until out of range and then back.
Its only about a minute or two tops but at 1msps it ended
up as 813 MB though.

https://drive.google.com/drive/folders/0B5dKo9igl8W4dWpiMmlEYWx3b0k?usp=sharing

>What you really care about is
>that there's a *tone* for about 8ms < t < 12ms where there is none else
>(to rule out detection of spurs and other interference), is that right?

Yes. The "tone" happens every 1.5 seconds and I want to catch as many
of those occurrences as possible as the receiver moves through space.
So for every 1.5 seconds I need to make a decision: was there one, or
not.

Note that the receiver is moving. So perhaps initially I may be well
out of range of the signal and get nothing. But as I move closer as
some point I will start picking up those pulses. The earlier and more
reliably I can pick them up the better.

I actually did spend some time looking at PLLs & the gnu radio block
at some point. However I readily admit to being a software person not
a DSP/Radio person. I have the day job to deal with today but I will
have a study of the PLL block given Andy's tips and report back.

Cheers
Dirk





On 14 March 2017 at 16:37, Andy Walls <address@hidden> wrote:
> On Tue, 2017-03-14 at 12:00 -0400, address@hidden
> wrote:
>> Date: Tue, 14 Mar 2017 15:49:44 +0100
>> From: Marcus M?ller <address@hidden>
>> To: address@hidden
>> Subject: Re: [Discuss-gnuradio] fast parallel filtering
>
>> Hi Andy,
>>
>> Dirk's problem was the low SNR in his recording, so he'd like to take
>> the shape of his pulse into consideration, but he doesn't know the
>> spectral position of the same ? Dirk, can you confirm?
>
> Ah, OK.
>
>
>> Dirk, thinking about this, the relation of the spectral uncertainty to
>> the bandwidth of the pulse would be interesting  ? can you refresh our
>> memory? I think the pulse was 10ms long (so, pretty long) but I forgot
>> how it looked like.
>>
>> Having slept about this a couple of times: What you really care about
>> is
>> that there's a *tone* for about 8ms < t < 12ms where there is none
>> else
>> (to rule out detection of spurs and other interference), is that
>> right?
>>
>> Maybe we've been focussing too much on filter-based detection. What if
>> we just *use* that feature of the signal being periodic for a duration
>> of t, and not else? A PLL would be able to "lock" on the tone (much
>> like
>> an FM Radio with a PLL will lock on the station's carrier), and if we
>> observe the phase error being limited for t, we can derive there was a
>> pulse.
>>
>> Andy, does that sound like a feasible approach?
>
> Yes.  With the added benefit of GNURadio's carrier tracking PLL actually
> performing part of the correlation operation for you, if it works out.
>
> So use the carrier_tracking_pll block.  If it locks onto a good signal,
> it will mix it down to DC.  Then all you need is an averaging filter,
> like the single pole IIR filter block or the moving average filter
> block, operating on the real part of that output.  That will act as a
> lock detector, and, I think, also completes a correlation operation, if
> you use a non-normalized moving average filter (since the carrier
> tracking PLL block gives you a point by point complex multiply with the
> conjugate of the complex carrier tone).
>
> You'll want to set the PLL allowed min and max frequency bounds
> properly; be careful since the input arguments have tricky units.
>
> You'll also want the loop bandwidth set pretty wide, since you want to
> lock-in rapidly.
>
> Also, if I'm thinking about this properly:
> To get faster lock in, you may want your frequency range of interest to
> be somewhere in between Fs/4 and Fs/2; and not near DC.  If you're
> within the lock-in range of the PLL, you'll lock within 1 cycle.  If
> you're in the pull-in range of the PLL, it can take many cycles to get
> into lock.
>
> Regards,
> Andy
>
>> Cheers,
>>
>> Marcus
>>
>>
>> On 14.03.2017 14:18, Andy Walls wrote:
>> > If there is no information on the signal, why not just do a straight
>> > energy detector:
>> >
>> > source -> complex bandpass filter -> complex to mag -> burst/energy
>> detector -> QT Time sink (triggering on start of burst tag)
>> >
>> > The complex bandpass filter just has to catch all the frequencies of
>> > interest.  (A complex bandpass filter does not have a symmetrical
>> image
>> > around DC.)
>> >
>> > The burst/energy detector has to detect some minimum number of time
>> > domain samples contiguously above some noise/signal threshold that
>> you
>> > set, and tag the pulse.  It also has to detect when the time domain
>> > samples fall below the threshold.  The burst/energy detector can
>> work
>> > with a preset noise/signal threshold, or you could periodically
>> > re-estimate that threshold.
>> >
>> > GNURadio does not have a stock block that does burst/energy
>> detection,
>> > so you'll have to find one on the 'Net or write one yourself.
>> >
>> > -Andy
>> >
>> >
>> > Dirk Gorissen wrote:
>> >> Hi Martin,
>> >>
>> >> The aim is to check for the existence of a 10ms pulse in the
>> incoming
>> >> radio signal (from an sdr). The thing is I only know the frequency
>> of
>> >> the pulse to within ~1khz.
>> >>
>> >> So a single filter in my case is: generate a synthetic pulse
>> (complex
>> >> sin wave) of the same length and with a frequency f. Then take the
>> >> reverse of the complex conjugate of this pulse to give me the taps
>> for
>> >> a FIR filter.
>> >>
>> >> Repeat the above for each frequency I want to check. E.g., 10
>> filters
>> >> each 100Hz apart.
>> >> Then I just take the magnitude of each filter output and push
>> through
>> >> a Max to get my final correlations.
>> >>
>> >> I can come up with an algorithm that tries to be clever and with
>> some
>> >> accounting tries to find what frequency matters most but I wouldnt
>> be
>> >> surprised if there is some theory you can use to do this more
>> >> efficiently or even in a single shot. But this is where Im unsure.
>> >>
>> >> Cheers
>> >> Dirk
>> >>
>> >>
>> >> On 14 March 2017 at 01:09, Martin Braun <address@hidden> wrote:
>> >>> How related are those filters? Is this a candidate for polyphase
>> DSP?
>> >>>
>> >>> -- M
>> >>>
>> >>> On 03/11/2017 02:01 PM, Dirk Gorissen wrote:
>> >>>>  Hi Marcus,
>> >>>>
>> >>>>  Sorry, I should have clarified. You may recall an earlier thread
>> from
>> >>>>  mine where Im looking to pick out a short pulse from background
>> noise
>> >>>>  but I dont know the exact frequency of the pulse. Thought I
>> would
>> >>>>  start a new thread with a clear, specific question.
>> >>>>
>> >>>>  There is an uncertainty of +/- 1khz. So I can define multiple
>> filters,
>> >>>>  correlating for different pulse frequencies across the 1khz
>> range. I
>> >>>>  can implement this with different filters running in parallel
>> but I
>> >>>>  was looking for a more flexible / efficient way.
>> >>>>
>> >>>>  If you think this is out of scope for this list no problem at
>> all,
>> >>>>  just let me know.
>> >>>>
>> >>>>  Cheers
>> >>>>  Dirk
>> >>>>
>> >>>>> On 11 March 2017 at 20:02, Marcus M?ller <address@hidden> wrote:
>> >>>>>> Hi Dirk,
>> >>>>>>
>> >>>>>> this is more of a general DSP question than a GNU Radio
>> question:
>> >>>>>>
>> >>>>>> How do these filters relate to each other?
>> >>>>>>
>> >>>>>> My gut feeling is that this gets a lot easier as soon as you
>> tell us why
>> >>>>>> you're doing this, i.e. for what purpose :)
>> >>>>>>
>> >>>>>> Best regards,
>> >>>>>> Marcus
>> >>>>>>
>> >>>>>> On 11.03.2017 19:28, Dirk Gorissen wrote:
>> >>>>>>> Hello all,
>> >>>>>>>
>> >>>>>>> Given a stream of samples I would like to apply n slightly
>> different
>> >>>>>>> filters to it with n being able to be chosen at runtime. Then
>> combine
>> >>>>>>> the results back to a single stream.
>> >>>>>>>
>> >>>>>>> As a test I built a flowgraph with the following chains in
>> parallel for n
>> >>>>>>> = 6
>> >>>>>>>
>> >>>>>>>                 | ->  decimating fir filter 1 -> complex to
>> mag ->    |
>> >>>>>>> stream -> | ->  decimating fir filter 2 -> complex to mag ->
>> |  -> Max
>> >>>>>>> -> ...
>> >>>>>>>                 |                                  ....
>> >>>>>>>                         |
>> >>>>>>>                 | ->  decimating fir filter n -> complex to
>> mag ->    |
>> >>>>>>>
>> >>>>>>> So the same stream is sent to each chain (decimation is 1) and
>> the
>> >>>>>>> output of each chain is pushed through a big Max block with 6
>> inputs.
>> >>>>>>>
>> >>>>>>> This works but not particularly elegant and a bit annoying to
>> change
>> >>>>>>> if I suddenly decide I want to change n. In particular it also
>> does
>> >>>>>>> not seem computationally efficient.
>> >>>>>>>
>> >>>>>>> What I would like is to replace the above by a single block
>> that
>> >>>>>>>
>> >>>>>>> - replicates the input n times
>> >>>>>>> - applies each filter on each replica
>> >>>>>>> - combines the output again to a single stream
>> >>>>>>> - have a tunable n parameter
>> >>>>>>> - is fast
>> >>>>>>>
>> >>>>>>> I did this with an Embedded python block doing essentially
>> this:
>> >>>>>>>
>> >>>>>>> for i in range(n):
>> >>>>>>>      out[i] = scipy.signal.lfilter(taps[i], 1, input)
>> >>>>>>>
>> >>>>>>> This is using exactly the same taps as in the chain case. This
>> works
>> >>>>>>> but the output is different and worse than what I get with the
>> >>>>>>> separate chains. As a test, instead of lfilter I tried:
>> >>>>>>>
>> >>>>>>>
>> gnuradio.filter.fir_filter_ccc(1,taps[i]).work(input[0],output)
>> >>>>>>>
>> >>>>>>> Thinking perhaps that is a closer replica. But couldnt get it
>> to work..
>> >>>>>>>
>> >>>>>>> I suspect there should be an easy / natural way of doing this
>> in
>> >>>>>>> gnuradio. Looked at the filter bank / channelliser blocks but
>> failed
>> >>>>>>> to get anywhere.
>> >>>>>>>
>> >>>>>>> So what is the best way forward to do this?
>> >>>>>>>
>> >>>>>>> Many thanks
>> >>>>>>> Dirk
>> >>>>>>>
>> >>>>>>> _______________________________________________
>> >>>>>>> Discuss-gnuradio mailing list
>> >>>>>>> address@hidden
>> >>>>>>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>> >>>>>>
>> >>>>>> _______________________________________________
>> >>>>>> Discuss-gnuradio mailing list
>> >>>>>> address@hidden
>> >>>>>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> _________________________________________
>> >>>>> Dr. Dirk Gorissen
>> >>>>> Mob: +44-7763-806-809
>> >>>>> Web: dirkgorissen.com
>> >>>>> Skype: dirk.gorissen
>> >>>>> Twitter  : twitter.com/dirkgor
>> >>>>> LinkedIn: linkedin.com/in/dirkgorissen
>> >>>>
>> >>>>
>> >>>
>> >>> _______________________________________________
>> >>> Discuss-gnuradio mailing list
>> >>> address@hidden
>> >>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>> >>
>> >>
>> >> --
>> >> _________________________________________
>> >> Dr. Dirk Gorissen
>> >> Mob: +44-7763-806-809
>> >> Web: dirkgorissen.com
>> >> Skype: dirk.gorissen
>> >> Twitter  : twitter.com/dirkgor
>> >> LinkedIn: linkedin.com/in/dirkgorissen
>> >
>> >
>> > _______________________________________________
>> > Discuss-gnuradio mailing list
>> > address@hidden
>> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>
>



-- 
_________________________________________
Dr. Dirk Gorissen
Mob: +44-7763-806-809
Web: dirkgorissen.com
Skype: dirk.gorissen
Twitter  : twitter.com/dirkgor
LinkedIn: linkedin.com/in/dirkgorissen