Re: [Discuss-gnuradio] fast parallel filtering

[Andy Walls]

Hi Dirk,

Thanks.

Try the attached flowgraph.  The pulse indicator output is a little crude
at the moment (it needs some latching persistence for a human to read),
but it does the job for a proof of concept.

Note that in the future, you really do want to tune such the the SDR's center
freq spike is not in the band of interest.  That will prevent false lock on the
DC spike by the PLL.

Regards,
Andy

On Thu, Mar 16, 2017 at 7:12 PM, Dirk Gorissen <address@hidden> wrote:
> Hi Andy,
>
> Very quickly collected some data off a different Tx. First at very
> close range, moving away, and then back again. It was quick and
> antenna was close to computer so perhaps quite noisy but should
> definitely have some strong pulses:
>
> https://drive.google.com/open?id=0B5dKo9igl8W4WkRkUGQzcVJsQnc
>
> Havent managed to look at the pll stuff yet. Aim to do so this weekend.
> Cheers
> Dirk
>
> On 16 March 2017 at 07:22, Dirk Gorissen <address@hidden> wrote:
>> Mmm, thats odd. Those settings should be correct, maybe something went
>> wrong with the recording :(  I'll try to check in between things at
>> work today else will do so tonight.
>> Thanks for taking the time to look though.
>>
>> On 16 March 2017 at 00:39, Andy Walls <address@hidden> wrote:
>>> Hi Dirk:
>>>
>>> In the IQ data file you provided I can seem to find any pulses of the
>>> nominal 10 msec length, no matter how I filter and rotate the
>>> spectrum.
>>> There is a lot of EMI, which looks like the intermodulation products
>>> of a continuously on guy who is drifting/chirping down in frequency.
>>>
>>> So could you please confirm or clarify the following:
>>>
>>> 1. The format of the binary data file.  I am assuming 32 bit float I
>>> and 32 bit float Q pairs, as that is what the GNURadio file sink would
>>> normally create.
>>> 2. The sample rate.  I am assuming 1 Msps.
>>> 3. The center freq.  I am assuming 150.22 MHz.
>>> 4. The pulse duration. I am assuming 10 milliseconds.
>>> 5. At what frequency offset, from the center frequency, should the
>>> pulse be at?  I'm assuming somewhere withing +/- 5 kHz of the center
>>> spike, but there are at least two EMI spikes in that range.
>>>
>>> Thanks.
>>>
>>> -Andy
>>>
>>> On Wed, Mar 15, 2017 at 5:23 AM, Dirk Gorissen <address@hidden> wrote:
>>>> Hi Andy, Marcus,
>>>>
>>>> Thanks very much for taking the time to think about this.
>>>>
>>>> Just to answer your questions:
>>>>
>>>>>Dirk's problem was the low SNR in his recording, so he'd like to take
>>>>>the shape of his pulse into consideration, but he doesn't know the
>>>>>spectral position of the same – Dirk, can you confirm?
>>>>
>>>> Yes, Im dealing with a very weak signal against quite a bit of
>>>> background noise so the more prior knowledge I can leverage the
>>>> better.
>>>>
>>>>>Dirk, thinking about this, the relation of the spectral uncertainty to
>>>>>the bandwidth of the pulse would be interesting  – can you refresh our
>>>>>memory? I think the pulse was 10ms long (so, pretty long) but I forgot
>>>>>how it looked like.
>>>>
>>>> Yes, 10ms long (this is pretty exact), occurring every 1.5 seconds, in
>>>> the 150MHz range. My input stream is coming from an SDR.
>>>> As an aside I actually have multiple transmitters each pulsing at
>>>> slightly different frequencies (e.g., 150.10, 150.22, ...) but Im
>>>> happy to treat them independently so you can ignore that and assume
>>>> there is just one.
>>>>
>>>> You can see pictures of the pulse (taken a while back, for a specific
>>>> tx frequency) here: https://goo.gl/photos/Y3Ea6kJo1ewrcDYM8
>>>>
>>>> Note though that this is taken with the transmitter very close to the
>>>> antenna so the signal is unrealistically strong. So its just to
>>>> illustrate.
>>>>
>>>> I also quickly captured a file with raw IQ samples (File sink) in case
>>>> anybody is interested. It starts with the transmitter very close to
>>>> antenna, moves progressively further until out of range and then back.
>>>> Its only about a minute or two tops but at 1msps it ended
>>>> up as 813 MB though.
>>>>
>>>> https://drive.google.com/drive/folders/0B5dKo9igl8W4dWpiMmlEYWx3b0k?usp=sharing
>>>>
>>>>>What you really care about is
>>>>>that there's a *tone* for about 8ms < t < 12ms where there is none else
>>>>>(to rule out detection of spurs and other interference), is that right?
>>>>
>>>> Yes. The "tone" happens every 1.5 seconds and I want to catch as many
>>>> of those occurrences as possible as the receiver moves through space.
>>>> So for every 1.5 seconds I need to make a decision: was there one, or
>>>> not.
>>>>
>>>> Note that the receiver is moving. So perhaps initially I may be well
>>>> out of range of the signal and get nothing. But as I move closer as
>>>> some point I will start picking up those pulses. The earlier and more
>>>> reliably I can pick them up the better.
>>>>
>>>> I actually did spend some time looking at PLLs & the gnu radio block
>>>> at some point. However I readily admit to being a software person not
>>>> a DSP/Radio person. I have the day job to deal with today but I will
>>>> have a study of the PLL block given Andy's tips and report back.
>>>>
>>>> Cheers
>>>> Dirk
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 14 March 2017 at 16:37, Andy Walls <address@hidden> wrote:
>>>>> On Tue, 2017-03-14 at 12:00 -0400, address@hidden
>>>>> wrote:
>>>>>> Date: Tue, 14 Mar 2017 15:49:44 +0100
>>>>>> From: Marcus M?ller <address@hidden>
>>>>>> To: address@hidden
>>>>>> Subject: Re: [Discuss-gnuradio] fast parallel filtering
>>>>>
>>>>>> Hi Andy,
>>>>>>
>>>>>> Dirk's problem was the low SNR in his recording, so he'd like to take
>>>>>> the shape of his pulse into consideration, but he doesn't know the
>>>>>> spectral position of the same ? Dirk, can you confirm?
>>>>>
>>>>> Ah, OK.
>>>>>
>>>>>
>>>>>> Dirk, thinking about this, the relation of the spectral uncertainty to
>>>>>> the bandwidth of the pulse would be interesting  ? can you refresh our
>>>>>> memory? I think the pulse was 10ms long (so, pretty long) but I forgot
>>>>>> how it looked like.
>>>>>>
>>>>>> Having slept about this a couple of times: What you really care about
>>>>>> is
>>>>>> that there's a *tone* for about 8ms < t < 12ms where there is none
>>>>>> else
>>>>>> (to rule out detection of spurs and other interference), is that
>>>>>> right?
>>>>>>
>>>>>> Maybe we've been focussing too much on filter-based detection. What if
>>>>>> we just *use* that feature of the signal being periodic for a duration
>>>>>> of t, and not else? A PLL would be able to "lock" on the tone (much
>>>>>> like
>>>>>> an FM Radio with a PLL will lock on the station's carrier), and if we
>>>>>> observe the phase error being limited for t, we can derive there was a
>>>>>> pulse.
>>>>>>
>>>>>> Andy, does that sound like a feasible approach?
>>>>>
>>>>> Yes.  With the added benefit of GNURadio's carrier tracking PLL actually
>>>>> performing part of the correlation operation for you, if it works out.
>>>>>
>>>>> So use the carrier_tracking_pll block.  If it locks onto a good signal,
>>>>> it will mix it down to DC.  Then all you need is an averaging filter,
>>>>> like the single pole IIR filter block or the moving average filter
>>>>> block, operating on the real part of that output.  That will act as a
>>>>> lock detector, and, I think, also completes a correlation operation, if
>>>>> you use a non-normalized moving average filter (since the carrier
>>>>> tracking PLL block gives you a point by point complex multiply with the
>>>>> conjugate of the complex carrier tone).
>>>>>
>>>>> You'll want to set the PLL allowed min and max frequency bounds
>>>>> properly; be careful since the input arguments have tricky units.
>>>>>
>>>>> You'll also want the loop bandwidth set pretty wide, since you want to
>>>>> lock-in rapidly.
>>>>>
>>>>> Also, if I'm thinking about this properly:
>>>>> To get faster lock in, you may want your frequency range of interest to
>>>>> be somewhere in between Fs/4 and Fs/2; and not near DC.  If you're
>>>>> within the lock-in range of the PLL, you'll lock within 1 cycle.  If
>>>>> you're in the pull-in range of the PLL, it can take many cycles to get
>>>>> into lock.
>>>>>
>>>>> Regards,
>>>>> Andy
>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Marcus
>>>>>>
>>>>>>
>>>>>> On 14.03.2017 14:18, Andy Walls wrote:
>>>>>> > If there is no information on the signal, why not just do a straight
>>>>>> > energy detector:
>>>>>> >
>>>>>> > source -> complex bandpass filter -> complex to mag -> burst/energy
>>>>>> detector -> QT Time sink (triggering on start of burst tag)
>>>>>> >
>>>>>> > The complex bandpass filter just has to catch all the frequencies of
>>>>>> > interest.  (A complex bandpass filter does not have a symmetrical
>>>>>> image
>>>>>> > around DC.)
>>>>>> >
>>>>>> > The burst/energy detector has to detect some minimum number of time
>>>>>> > domain samples contiguously above some noise/signal threshold that
>>>>>> you
>>>>>> > set, and tag the pulse.  It also has to detect when the time domain
>>>>>> > samples fall below the threshold.  The burst/energy detector can
>>>>>> work
>>>>>> > with a preset noise/signal threshold, or you could periodically
>>>>>> > re-estimate that threshold.
>>>>>> >
>>>>>> > GNURadio does not have a stock block that does burst/energy
>>>>>> detection,
>>>>>> > so you'll have to find one on the 'Net or write one yourself.
>>>>>> >
>>>>>> > -Andy
>>>>>> >
>>>>>> >
>>>>>> > Dirk Gorissen wrote:
>>>>>> >> Hi Martin,
>>>>>> >>
>>>>>> >> The aim is to check for the existence of a 10ms pulse in the
>>>>>> incoming
>>>>>> >> radio signal (from an sdr). The thing is I only know the frequency
>>>>>> of
>>>>>> >> the pulse to within ~1khz.
>>>>>> >>
>>>>>> >> So a single filter in my case is: generate a synthetic pulse
>>>>>> (complex
>>>>>> >> sin wave) of the same length and with a frequency f. Then take the
>>>>>> >> reverse of the complex conjugate of this pulse to give me the taps
>>>>>> for
>>>>>> >> a FIR filter.
>>>>>> >>
>>>>>> >> Repeat the above for each frequency I want to check. E.g., 10
>>>>>> filters
>>>>>> >> each 100Hz apart.
>>>>>> >> Then I just take the magnitude of each filter output and push
>>>>>> through
>>>>>> >> a Max to get my final correlations.
>>>>>> >>
>>>>>> >> I can come up with an algorithm that tries to be clever and with
>>>>>> some
>>>>>> >> accounting tries to find what frequency matters most but I wouldnt
>>>>>> be
>>>>>> >> surprised if there is some theory you can use to do this more
>>>>>> >> efficiently or even in a single shot. But this is where Im unsure.
>>>>>> >>
>>>>>> >> Cheers
>>>>>> >> Dirk
>>>>>> >>
>>>>>> >>
>>>>>> >> On 14 March 2017 at 01:09, Martin Braun <address@hidden> wrote:
>>>>>> >>> How related are those filters? Is this a candidate for polyphase
>>>>>> DSP?
>>>>>> >>>
>>>>>> >>> -- M
>>>>>> >>>
>>>>>> >>> On 03/11/2017 02:01 PM, Dirk Gorissen wrote:
>>>>>> >>>>  Hi Marcus,
>>>>>> >>>>
>>>>>> >>>>  Sorry, I should have clarified. You may recall an earlier thread
>>>>>> from
>>>>>> >>>>  mine where Im looking to pick out a short pulse from background
>>>>>> noise
>>>>>> >>>>  but I dont know the exact frequency of the pulse. Thought I
>>>>>> would
>>>>>> >>>>  start a new thread with a clear, specific question.
>>>>>> >>>>
>>>>>> >>>>  There is an uncertainty of +/- 1khz. So I can define multiple
>>>>>> filters,
>>>>>> >>>>  correlating for different pulse frequencies across the 1khz
>>>>>> range. I
>>>>>> >>>>  can implement this with different filters running in parallel
>>>>>> but I
>>>>>> >>>>  was looking for a more flexible / efficient way.
>>>>>> >>>>
>>>>>> >>>>  If you think this is out of scope for this list no problem at
>>>>>> all,
>>>>>> >>>>  just let me know.
>>>>>> >>>>
>>>>>> >>>>  Cheers
>>>>>> >>>>  Dirk
>>>>>> >>>>
>>>>>> >>>>> On 11 March 2017 at 20:02, Marcus M?ller <address@hidden> wrote:
>>>>>> >>>>>> Hi Dirk,
>>>>>> >>>>>>
>>>>>> >>>>>> this is more of a general DSP question than a GNU Radio
>>>>>> question:
>>>>>> >>>>>>
>>>>>> >>>>>> How do these filters relate to each other?
>>>>>> >>>>>>
>>>>>> >>>>>> My gut feeling is that this gets a lot easier as soon as you
>>>>>> tell us why
>>>>>> >>>>>> you're doing this, i.e. for what purpose :)
>>>>>> >>>>>>
>>>>>> >>>>>> Best regards,
>>>>>> >>>>>> Marcus
>>>>>> >>>>>>
>>>>>> >>>>>> On 11.03.2017 19:28, Dirk Gorissen wrote:
>>>>>> >>>>>>> Hello all,
>>>>>> >>>>>>>
>>>>>> >>>>>>> Given a stream of samples I would like to apply n slightly
>>>>>> different
>>>>>> >>>>>>> filters to it with n being able to be chosen at runtime. Then
>>>>>> combine
>>>>>> >>>>>>> the results back to a single stream.
>>>>>> >>>>>>>
>>>>>> >>>>>>> As a test I built a flowgraph with the following chains in
>>>>>> parallel for n
>>>>>> >>>>>>> = 6
>>>>>> >>>>>>>
>>>>>> >>>>>>>                 | ->  decimating fir filter 1 -> complex to
>>>>>> mag ->    |
>>>>>> >>>>>>> stream -> | ->  decimating fir filter 2 -> complex to mag ->
>>>>>> |  -> Max
>>>>>> >>>>>>> -> ...
>>>>>> >>>>>>>                 |                                  ....
>>>>>> >>>>>>>                         |
>>>>>> >>>>>>>                 | ->  decimating fir filter n -> complex to
>>>>>> mag ->    |
>>>>>> >>>>>>>
>>>>>> >>>>>>> So the same stream is sent to each chain (decimation is 1) and
>>>>>> the
>>>>>> >>>>>>> output of each chain is pushed through a big Max block with 6
>>>>>> inputs.
>>>>>> >>>>>>>
>>>>>> >>>>>>> This works but not particularly elegant and a bit annoying to
>>>>>> change
>>>>>> >>>>>>> if I suddenly decide I want to change n. In particular it also
>>>>>> does
>>>>>> >>>>>>> not seem computationally efficient.
>>>>>> >>>>>>>
>>>>>> >>>>>>> What I would like is to replace the above by a single block
>>>>>> that
>>>>>> >>>>>>>
>>>>>> >>>>>>> - replicates the input n times
>>>>>> >>>>>>> - applies each filter on each replica
>>>>>> >>>>>>> - combines the output again to a single stream
>>>>>> >>>>>>> - have a tunable n parameter
>>>>>> >>>>>>> - is fast
>>>>>> >>>>>>>
>>>>>> >>>>>>> I did this with an Embedded python block doing essentially
>>>>>> this:
>>>>>> >>>>>>>
>>>>>> >>>>>>> for i in range(n):
>>>>>> >>>>>>>      out[i] = scipy.signal.lfilter(taps[i], 1, input)
>>>>>> >>>>>>>
>>>>>> >>>>>>> This is using exactly the same taps as in the chain case. This
>>>>>> works
>>>>>> >>>>>>> but the output is different and worse than what I get with the
>>>>>> >>>>>>> separate chains. As a test, instead of lfilter I tried:
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> gnuradio.filter.fir_filter_ccc(1,taps[i]).work(input[0],output)
>>>>>> >>>>>>>
>>>>>> >>>>>>> Thinking perhaps that is a closer replica. But couldnt get it
>>>>>> to work..
>>>>>> >>>>>>>
>>>>>> >>>>>>> I suspect there should be an easy / natural way of doing this
>>>>>> in
>>>>>> >>>>>>> gnuradio. Looked at the filter bank / channelliser blocks but
>>>>>> failed
>>>>>> >>>>>>> to get anywhere.
>>>>>> >>>>>>>
>>>>>> >>>>>>> So what is the best way forward to do this?
>>>>>> >>>>>>>
>>>>>> >>>>>>> Many thanks
>>>>>> >>>>>>> Dirk
>>>>>> >>>>>>>
>>>>>> >>>>>>> _______________________________________________
>>>>>> >>>>>>> Discuss-gnuradio mailing list
>>>>>> >>>>>>> address@hidden
>>>>>> >>>>>>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>>>>> >>>>>>
>>>>>> >>>>>> _______________________________________________
>>>>>> >>>>>> Discuss-gnuradio mailing list
>>>>>> >>>>>> address@hidden
>>>>>> >>>>>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> --
>>>>>> >>>>> _________________________________________
>>>>>> >>>>> Dr. Dirk Gorissen
>>>>>> >>>>> Mob: +44-7763-806-809
>>>>>> >>>>> Web: dirkgorissen.com
>>>>>> >>>>> Skype: dirk.gorissen
>>>>>> >>>>> Twitter  : twitter.com/dirkgor
>>>>>> >>>>> LinkedIn: linkedin.com/in/dirkgorissen
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>> _______________________________________________
>>>>>> >>> Discuss-gnuradio mailing list
>>>>>> >>> address@hidden
>>>>>> >>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> _________________________________________
>>>>>> >> Dr. Dirk Gorissen
>>>>>> >> Mob: +44-7763-806-809
>>>>>> >> Web: dirkgorissen.com
>>>>>> >> Skype: dirk.gorissen
>>>>>> >> Twitter  : twitter.com/dirkgor
>>>>>> >> LinkedIn: linkedin.com/in/dirkgorissen
>>>>>> >
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > Discuss-gnuradio mailing list
>>>>>> > address@hidden
>>>>>> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> _________________________________________
>>>> Dr. Dirk Gorissen
>>>> Mob: +44-7763-806-809
>>>> Web: dirkgorissen.com
>>>> Skype: dirk.gorissen
>>>> Twitter  : twitter.com/dirkgor
>>>> LinkedIn: linkedin.com/in/dirkgorissen
>>
>>
>>
>> --
>> _________________________________________
>> Dr. Dirk Gorissen
>> Mob: +44-7763-806-809
>> Web: dirkgorissen.com
>> Skype: dirk.gorissen
>> Twitter  : twitter.com/dirkgor
>> LinkedIn: linkedin.com/in/dirkgorissen
>
>
>
> --
> _________________________________________
> Dr. Dirk Gorissen
> Mob: +44-7763-806-809
> Web: dirkgorissen.com
> Skype: dirk.gorissen
> Twitter  : twitter.com/dirkgor
> LinkedIn: linkedin.com/in/dirkgorissen

implant_pulse_detect.grc
Description: Binary data