Re: [Discuss-gnuradio] fast parallel filtering

[Dirk Gorissen]

Mmm, thats odd. Those settings should be correct, maybe something went
wrong with the recording :(  I'll try to check in between things at
work today else will do so tonight.
Thanks for taking the time to look though.

On 16 March 2017 at 00:39, Andy Walls <address@hidden> wrote:
> Hi Dirk:
>
> In the IQ data file you provided I can seem to find any pulses of the
> nominal 10 msec length, no matter how I filter and rotate the
> spectrum.
> There is a lot of EMI, which looks like the intermodulation products
> of a continuously on guy who is drifting/chirping down in frequency.
>
> So could you please confirm or clarify the following:
>
> 1. The format of the binary data file.  I am assuming 32 bit float I
> and 32 bit float Q pairs, as that is what the GNURadio file sink would
> normally create.
> 2. The sample rate.  I am assuming 1 Msps.
> 3. The center freq.  I am assuming 150.22 MHz.
> 4. The pulse duration. I am assuming 10 milliseconds.
> 5. At what frequency offset, from the center frequency, should the
> pulse be at?  I'm assuming somewhere withing +/- 5 kHz of the center
> spike, but there are at least two EMI spikes in that range.
>
> Thanks.
>
> -Andy
>
> On Wed, Mar 15, 2017 at 5:23 AM, Dirk Gorissen <address@hidden> wrote:
>> Hi Andy, Marcus,
>>
>> Thanks very much for taking the time to think about this.
>>
>> Just to answer your questions:
>>
>>>Dirk's problem was the low SNR in his recording, so he'd like to take
>>>the shape of his pulse into consideration, but he doesn't know the
>>>spectral position of the same – Dirk, can you confirm?
>>
>> Yes, Im dealing with a very weak signal against quite a bit of
>> background noise so the more prior knowledge I can leverage the
>> better.
>>
>>>Dirk, thinking about this, the relation of the spectral uncertainty to
>>>the bandwidth of the pulse would be interesting  – can you refresh our
>>>memory? I think the pulse was 10ms long (so, pretty long) but I forgot
>>>how it looked like.
>>
>> Yes, 10ms long (this is pretty exact), occurring every 1.5 seconds, in
>> the 150MHz range. My input stream is coming from an SDR.
>> As an aside I actually have multiple transmitters each pulsing at
>> slightly different frequencies (e.g., 150.10, 150.22, ...) but Im
>> happy to treat them independently so you can ignore that and assume
>> there is just one.
>>
>> You can see pictures of the pulse (taken a while back, for a specific
>> tx frequency) here: https://goo.gl/photos/Y3Ea6kJo1ewrcDYM8
>>
>> Note though that this is taken with the transmitter very close to the
>> antenna so the signal is unrealistically strong. So its just to
>> illustrate.
>>
>> I also quickly captured a file with raw IQ samples (File sink) in case
>> anybody is interested. It starts with the transmitter very close to
>> antenna, moves progressively further until out of range and then back.
>> Its only about a minute or two tops but at 1msps it ended
>> up as 813 MB though.
>>
>> https://drive.google.com/drive/folders/0B5dKo9igl8W4dWpiMmlEYWx3b0k?usp=sharing
>>
>>>What you really care about is
>>>that there's a *tone* for about 8ms < t < 12ms where there is none else
>>>(to rule out detection of spurs and other interference), is that right?
>>
>> Yes. The "tone" happens every 1.5 seconds and I want to catch as many
>> of those occurrences as possible as the receiver moves through space.
>> So for every 1.5 seconds I need to make a decision: was there one, or
>> not.
>>
>> Note that the receiver is moving. So perhaps initially I may be well
>> out of range of the signal and get nothing. But as I move closer as
>> some point I will start picking up those pulses. The earlier and more
>> reliably I can pick them up the better.
>>
>> I actually did spend some time looking at PLLs & the gnu radio block
>> at some point. However I readily admit to being a software person not
>> a DSP/Radio person. I have the day job to deal with today but I will
>> have a study of the PLL block given Andy's tips and report back.
>>
>> Cheers
>> Dirk
>>
>>
>>
>>
>>
>> On 14 March 2017 at 16:37, Andy Walls <address@hidden> wrote:
>>> On Tue, 2017-03-14 at 12:00 -0400, address@hidden
>>> wrote:
>>>> Date: Tue, 14 Mar 2017 15:49:44 +0100
>>>> From: Marcus M?ller <address@hidden>
>>>> To: address@hidden
>>>> Subject: Re: [Discuss-gnuradio] fast parallel filtering
>>>
>>>> Hi Andy,
>>>>
>>>> Dirk's problem was the low SNR in his recording, so he'd like to take
>>>> the shape of his pulse into consideration, but he doesn't know the
>>>> spectral position of the same ? Dirk, can you confirm?
>>>
>>> Ah, OK.
>>>
>>>
>>>> Dirk, thinking about this, the relation of the spectral uncertainty to
>>>> the bandwidth of the pulse would be interesting  ? can you refresh our
>>>> memory? I think the pulse was 10ms long (so, pretty long) but I forgot
>>>> how it looked like.
>>>>
>>>> Having slept about this a couple of times: What you really care about
>>>> is
>>>> that there's a *tone* for about 8ms < t < 12ms where there is none
>>>> else
>>>> (to rule out detection of spurs and other interference), is that
>>>> right?
>>>>
>>>> Maybe we've been focussing too much on filter-based detection. What if
>>>> we just *use* that feature of the signal being periodic for a duration
>>>> of t, and not else? A PLL would be able to "lock" on the tone (much
>>>> like
>>>> an FM Radio with a PLL will lock on the station's carrier), and if we
>>>> observe the phase error being limited for t, we can derive there was a
>>>> pulse.
>>>>
>>>> Andy, does that sound like a feasible approach?
>>>
>>> Yes.  With the added benefit of GNURadio's carrier tracking PLL actually
>>> performing part of the correlation operation for you, if it works out.
>>>
>>> So use the carrier_tracking_pll block.  If it locks onto a good signal,
>>> it will mix it down to DC.  Then all you need is an averaging filter,
>>> like the single pole IIR filter block or the moving average filter
>>> block, operating on the real part of that output.  That will act as a
>>> lock detector, and, I think, also completes a correlation operation, if
>>> you use a non-normalized moving average filter (since the carrier
>>> tracking PLL block gives you a point by point complex multiply with the
>>> conjugate of the complex carrier tone).
>>>
>>> You'll want to set the PLL allowed min and max frequency bounds
>>> properly; be careful since the input arguments have tricky units.
>>>
>>> You'll also want the loop bandwidth set pretty wide, since you want to
>>> lock-in rapidly.
>>>
>>> Also, if I'm thinking about this properly:
>>> To get faster lock in, you may want your frequency range of interest to
>>> be somewhere in between Fs/4 and Fs/2; and not near DC.  If you're
>>> within the lock-in range of the PLL, you'll lock within 1 cycle.  If
>>> you're in the pull-in range of the PLL, it can take many cycles to get
>>> into lock.
>>>
>>> Regards,
>>> Andy
>>>
>>>> Cheers,
>>>>
>>>> Marcus
>>>>
>>>>
>>>> On 14.03.2017 14:18, Andy Walls wrote:
>>>> > If there is no information on the signal, why not just do a straight
>>>> > energy detector:
>>>> >
>>>> > source -> complex bandpass filter -> complex to mag -> burst/energy
>>>> detector -> QT Time sink (triggering on start of burst tag)
>>>> >
>>>> > The complex bandpass filter just has to catch all the frequencies of
>>>> > interest.  (A complex bandpass filter does not have a symmetrical
>>>> image
>>>> > around DC.)
>>>> >
>>>> > The burst/energy detector has to detect some minimum number of time
>>>> > domain samples contiguously above some noise/signal threshold that
>>>> you
>>>> > set, and tag the pulse.  It also has to detect when the time domain
>>>> > samples fall below the threshold.  The burst/energy detector can
>>>> work
>>>> > with a preset noise/signal threshold, or you could periodically
>>>> > re-estimate that threshold.
>>>> >
>>>> > GNURadio does not have a stock block that does burst/energy
>>>> detection,
>>>> > so you'll have to find one on the 'Net or write one yourself.
>>>> >
>>>> > -Andy
>>>> >
>>>> >
>>>> > Dirk Gorissen wrote:
>>>> >> Hi Martin,
>>>> >>
>>>> >> The aim is to check for the existence of a 10ms pulse in the
>>>> incoming
>>>> >> radio signal (from an sdr). The thing is I only know the frequency
>>>> of
>>>> >> the pulse to within ~1khz.
>>>> >>
>>>> >> So a single filter in my case is: generate a synthetic pulse
>>>> (complex
>>>> >> sin wave) of the same length and with a frequency f. Then take the
>>>> >> reverse of the complex conjugate of this pulse to give me the taps
>>>> for
>>>> >> a FIR filter.
>>>> >>
>>>> >> Repeat the above for each frequency I want to check. E.g., 10
>>>> filters
>>>> >> each 100Hz apart.
>>>> >> Then I just take the magnitude of each filter output and push
>>>> through
>>>> >> a Max to get my final correlations.
>>>> >>
>>>> >> I can come up with an algorithm that tries to be clever and with
>>>> some
>>>> >> accounting tries to find what frequency matters most but I wouldnt
>>>> be
>>>> >> surprised if there is some theory you can use to do this more
>>>> >> efficiently or even in a single shot. But this is where Im unsure.
>>>> >>
>>>> >> Cheers
>>>> >> Dirk
>>>> >>
>>>> >>
>>>> >> On 14 March 2017 at 01:09, Martin Braun <address@hidden> wrote:
>>>> >>> How related are those filters? Is this a candidate for polyphase
>>>> DSP?
>>>> >>>
>>>> >>> -- M
>>>> >>>
>>>> >>> On 03/11/2017 02:01 PM, Dirk Gorissen wrote:
>>>> >>>>  Hi Marcus,
>>>> >>>>
>>>> >>>>  Sorry, I should have clarified. You may recall an earlier thread
>>>> from
>>>> >>>>  mine where Im looking to pick out a short pulse from background
>>>> noise
>>>> >>>>  but I dont know the exact frequency of the pulse. Thought I
>>>> would
>>>> >>>>  start a new thread with a clear, specific question.
>>>> >>>>
>>>> >>>>  There is an uncertainty of +/- 1khz. So I can define multiple
>>>> filters,
>>>> >>>>  correlating for different pulse frequencies across the 1khz
>>>> range. I
>>>> >>>>  can implement this with different filters running in parallel
>>>> but I
>>>> >>>>  was looking for a more flexible / efficient way.
>>>> >>>>
>>>> >>>>  If you think this is out of scope for this list no problem at
>>>> all,
>>>> >>>>  just let me know.
>>>> >>>>
>>>> >>>>  Cheers
>>>> >>>>  Dirk
>>>> >>>>
>>>> >>>>> On 11 March 2017 at 20:02, Marcus M?ller <address@hidden> wrote:
>>>> >>>>>> Hi Dirk,
>>>> >>>>>>
>>>> >>>>>> this is more of a general DSP question than a GNU Radio
>>>> question:
>>>> >>>>>>
>>>> >>>>>> How do these filters relate to each other?
>>>> >>>>>>
>>>> >>>>>> My gut feeling is that this gets a lot easier as soon as you
>>>> tell us why
>>>> >>>>>> you're doing this, i.e. for what purpose :)
>>>> >>>>>>
>>>> >>>>>> Best regards,
>>>> >>>>>> Marcus
>>>> >>>>>>
>>>> >>>>>> On 11.03.2017 19:28, Dirk Gorissen wrote:
>>>> >>>>>>> Hello all,
>>>> >>>>>>>
>>>> >>>>>>> Given a stream of samples I would like to apply n slightly
>>>> different
>>>> >>>>>>> filters to it with n being able to be chosen at runtime. Then
>>>> combine
>>>> >>>>>>> the results back to a single stream.
>>>> >>>>>>>
>>>> >>>>>>> As a test I built a flowgraph with the following chains in
>>>> parallel for n
>>>> >>>>>>> = 6
>>>> >>>>>>>
>>>> >>>>>>>                 | ->  decimating fir filter 1 -> complex to
>>>> mag ->    |
>>>> >>>>>>> stream -> | ->  decimating fir filter 2 -> complex to mag ->
>>>> |  -> Max
>>>> >>>>>>> -> ...
>>>> >>>>>>>                 |                                  ....
>>>> >>>>>>>                         |
>>>> >>>>>>>                 | ->  decimating fir filter n -> complex to
>>>> mag ->    |
>>>> >>>>>>>
>>>> >>>>>>> So the same stream is sent to each chain (decimation is 1) and
>>>> the
>>>> >>>>>>> output of each chain is pushed through a big Max block with 6
>>>> inputs.
>>>> >>>>>>>
>>>> >>>>>>> This works but not particularly elegant and a bit annoying to
>>>> change
>>>> >>>>>>> if I suddenly decide I want to change n. In particular it also
>>>> does
>>>> >>>>>>> not seem computationally efficient.
>>>> >>>>>>>
>>>> >>>>>>> What I would like is to replace the above by a single block
>>>> that
>>>> >>>>>>>
>>>> >>>>>>> - replicates the input n times
>>>> >>>>>>> - applies each filter on each replica
>>>> >>>>>>> - combines the output again to a single stream
>>>> >>>>>>> - have a tunable n parameter
>>>> >>>>>>> - is fast
>>>> >>>>>>>
>>>> >>>>>>> I did this with an Embedded python block doing essentially
>>>> this:
>>>> >>>>>>>
>>>> >>>>>>> for i in range(n):
>>>> >>>>>>>      out[i] = scipy.signal.lfilter(taps[i], 1, input)
>>>> >>>>>>>
>>>> >>>>>>> This is using exactly the same taps as in the chain case. This
>>>> works
>>>> >>>>>>> but the output is different and worse than what I get with the
>>>> >>>>>>> separate chains. As a test, instead of lfilter I tried:
>>>> >>>>>>>
>>>> >>>>>>>
>>>> gnuradio.filter.fir_filter_ccc(1,taps[i]).work(input[0],output)
>>>> >>>>>>>
>>>> >>>>>>> Thinking perhaps that is a closer replica. But couldnt get it
>>>> to work..
>>>> >>>>>>>
>>>> >>>>>>> I suspect there should be an easy / natural way of doing this
>>>> in
>>>> >>>>>>> gnuradio. Looked at the filter bank / channelliser blocks but
>>>> failed
>>>> >>>>>>> to get anywhere.
>>>> >>>>>>>
>>>> >>>>>>> So what is the best way forward to do this?
>>>> >>>>>>>
>>>> >>>>>>> Many thanks
>>>> >>>>>>> Dirk
>>>> >>>>>>>
>>>> >>>>>>> _______________________________________________
>>>> >>>>>>> Discuss-gnuradio mailing list
>>>> >>>>>>> address@hidden
>>>> >>>>>>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>>> >>>>>>
>>>> >>>>>> _______________________________________________
>>>> >>>>>> Discuss-gnuradio mailing list
>>>> >>>>>> address@hidden
>>>> >>>>>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> --
>>>> >>>>> _________________________________________
>>>> >>>>> Dr. Dirk Gorissen
>>>> >>>>> Mob: +44-7763-806-809
>>>> >>>>> Web: dirkgorissen.com
>>>> >>>>> Skype: dirk.gorissen
>>>> >>>>> Twitter  : twitter.com/dirkgor
>>>> >>>>> LinkedIn: linkedin.com/in/dirkgorissen
>>>> >>>>
>>>> >>>>
>>>> >>>
>>>> >>> _______________________________________________
>>>> >>> Discuss-gnuradio mailing list
>>>> >>> address@hidden
>>>> >>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>>> >>
>>>> >>
>>>> >> --
>>>> >> _________________________________________
>>>> >> Dr. Dirk Gorissen
>>>> >> Mob: +44-7763-806-809
>>>> >> Web: dirkgorissen.com
>>>> >> Skype: dirk.gorissen
>>>> >> Twitter  : twitter.com/dirkgor
>>>> >> LinkedIn: linkedin.com/in/dirkgorissen
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Discuss-gnuradio mailing list
>>>> > address@hidden
>>>> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>>
>>>
>>
>>
>>
>> --
>> _________________________________________
>> Dr. Dirk Gorissen
>> Mob: +44-7763-806-809
>> Web: dirkgorissen.com
>> Skype: dirk.gorissen
>> Twitter  : twitter.com/dirkgor
>> LinkedIn: linkedin.com/in/dirkgorissen



-- 
_________________________________________
Dr. Dirk Gorissen
Mob: +44-7763-806-809
Web: dirkgorissen.com
Skype: dirk.gorissen
Twitter  : twitter.com/dirkgor
LinkedIn: linkedin.com/in/dirkgorissen