Re: [Discuss-gnuradio] intermittent pulse detection

[Dirk Gorissen]

Hi EJ, Vitt,

Thanks again for the pointers...

So I did as EJ suggested and implemented a FIR block that does a cross
correlation with the pulse sequence. The difference does not seem
massive but it does improve things a bit. The time plot is smoother
and peaks seem cleaner. Some further smoothing might be useful if
things get noisier.

>You will also need to roughly match the frequency value of the burst you are 
>trying to acquire;
>so you may want multiple FIR filters, each correlating against a sine wave of 
>a different frequency.

Some experimentation shows that my template can be off by up to about
100Hz, at that point peaks start to drown in the noise. So if my
uncertainty is 1khz and I would like full coverage of that bandwidth
that would mean that I need 10 filters, each 100Hz apart. As an
experiment I created 4  <FIR filter, complex->mag^2> chains, each 50Hz
apart, and fed them through a Max block. This seemed to work as
expected but this is where Im starting to worry about computational
cost. Is there an efficient way of applying 10 filters in parallel and
is a Max a sensible way of combining them?

Alternatively I can come up with an algorithm that tries to
create/apply filters dynamically based on where I think the pulse
seems to be. But if I can keep it simple that would be preferred.

Would also be nice if I could exploit the regular timing. Again I
could come up with some algorithm that decided when it has detected a
real peak and then only looks for them in a small window around the
expected next occurence. However, I assume people have thought about
this kind of problem and something exists (?). The only thing that
would make me nervous is that it would require all clocks to remain
stable and gnuradio not running out of sync (e.g., due to increased
load). How reliable is GR timing generally?

>in my experience SDR is a nice stuff but sometimes analog tech can help: try 
>to lower total NF using a tuned LNA in front of you
>receiver... RTL dongles or other sampler are a bit deaf and noisy, and your 
>signal seems very feeble.

Yes it is pretty weak. I do have an lna4all but it didn't seem to make
much difference. I also have an analog filter (140-160mhz range) but
not as narrow as I would like (know of any?) and also did not seem to
change much. Though it would need more rigorous testing I admit.

I know that SDR# contains various AGC and noise reduction algorithms.
Was wondering if any of those have analogues in GR or any such
routines I should be looking at specifically. Any tips welcome.

Cheers
Dirk

On 10 February 2017 at 11:14, Vitt Benv <address@hidden> wrote:
> Hi Dirk,
> in my experience SDR is a nice stuff but sometimes analog tech can help: try
> to lower total NF using a tuned LNA in front of you receiver... RTL dongles
> or other sampler are a bit deaf and noisy, and your signal seems very
> feeble.
>
> ... my 2 eurocent... !
>
> Victor
>
> 2017-02-10 0:01 GMT+01:00 Dirk Gorissen <address@hidden>:
>>
>> Hi Marcus,
>>
>> Thanks again. And yes, Id be happy to do a writeup if I get things
>> working with GnuRadio. I did a writeup of the first version I did of
>> my project (*) and happy to do a part two of the improved version when
>> finished (asap). Replies inline.
>>
>> >Well, if you have a let's say 2 kHz uncertainty in the frequency of your
>> > pulse, I'd really start very simple. Looking at your plots, I think we can
>> > sufficiently suppress the noise simply by low-pass filtering:
>>
>> Just note that the plots are posted are with the transmitter pretty
>> much sitting next to the antenna so this is a *very* strong signal, in
>> reality its much weaker.
>>
>> >* Use a Low Pass filter (real taps) with a cutoff of 1 kHz – that will
>> > let everything from -1 kHz to +1 kHz around your center frequency through.
>> > You can use the "freq xlating FIR" block to first shift by a given 
>> > frequency
>> > and >then apply the filter; might make things easier.
>> >* Detect power by putting your complex samples through a complex mag^2
>> > block.
>>
>> Thanks. I did this and the following observations
>> - I went SDR -> Freq Xlating FIR filter with 4x decimation -> low pass
>> filter block with 128x decimation and 1khz cutoff -> complex to mag2
>>
>> - The Freq Xlating block was giving me an error unless I filled in
>> something for the Tap, some googleing led me to fill in
>> firdes.low_pass_2 and I just used the same filter settings as the
>> filter block. That made it happy but to be honest Im not sure why and
>> if that was the right thing to do.
>>
>> - I did the decimation somewhat arbitrarily in two steps as I read
>> somewhere that is better computationally, perhaps it should be more
>> even though
>>
>> - With the above I could see the peaks (yay!) but it gets harder as
>> the signal gets weaker. I plugged in the peak detector block (peak
>> detector 2 seems broken btw) but found its usage very unintuitive and
>> I didnt get it to mark the peaks in the way I wanted. Even though they
>> were pretty big it wouldn't catch many of them, but perhaps its a
>> plotting refresh rate thing. Its not clear to me over what window the
>> average is taken (I was expecting that as an option).
>>
>> - Also, Im not 100% sure what sample rate I should be setting (or how
>> to count number of samples (e.g., peak look ahead)) in blocks that are
>> downstream from decimation. If the source sample rate is 1msps, and I
>> have decimated by 4x, say, that means I have 4x less samples coming
>> through per second so I should be setting 0.250msps as the sample rate
>> in downstream blocks (filters, display, fft, ...) right?
>>
>> >If the output of that looks somewhat OK, then you could e.g. low-pass
>> > filter the result to get rid of noise pulses that are too short, for 
>> > example
>> > :)
>>
>> Ok, so low-pass filter the real valued mag^2 values. I tried this but
>> was not obvious how to choose the right cutoff frequency & transition
>> width other than just trial and error. Also, as the signal gets weaker
>> the peaks get narrower.
>>
>> I still feel like I should be cross correlating with the known signal,
>> or some kind of matched filter setup, or exploiting the regularity of
>> the pulse somehow to give me more sensitivity, but unsure how best to
>> proceed there. I stumbled across a post about the correlate_and_sync
>> block. That seemed somewhat similar to my problem (though I guess I
>> have no preamble) but raised more questions than it solved.
>>
>> >By the way, it might be cool to put up a short recording (ie. a file
>> > generated using the GNU Radio file sink) somewhere (zipped, e.g. on Google
>> > Drive). I haven't worked with that myself, but maybe putting it on
>> >SigIDwiki[1] might be beneficial for people encountering the wildlife
>> > tracker signal later on.
>>
>> Good idea. I quickly captured a file with raw IQ samples (File sink)
>> in case anybody is interested. It starts with the transmitter very
>> close to antenna, moves progressively further until out of range and
>> then back. Its only about a minute or two tops but at 1msps it ended
>> up as 813 MB though.
>>
>>
>> https://drive.google.com/drive/folders/0B5dKo9igl8W4dWpiMmlEYWx3b0k?usp=sharing
>>
>> >We surely could use a few more "oh, GNU Radio is awesome because it works
>> > for everyone" articles (like [2]), but what we far more severely lack is
>> > stories of people that actually made progress, but also struggled,
>> >because they didn't know GR for years before writing a blog post. One of
>> > the problems that we face (and that was a very prominent theme at the Panel
>> > at FOSDEM [3]) is that we don't really know how to help people
>> >that are just approaching GNU Radio, and aren't already familiar with a
>> > lot of the stuff that we've gotten used to.
>>
>> Happy to do that. Generally I have found companion very robust and
>> works well. I love the python export/integration. Im just slightly
>> concerned about performance on a low spec machine but Ill cross that
>> bridge when I get to it. Main thing for me is that I find many
>> explanations for blocks very concise and without enough radio
>> background you feel a bit lost as to what exactly to use when or what
>> the options mean. Also, silly question but I am still confused as to
>> what a "tap" is. I googled around the docs hoping to find a page "what
>> is a tap" but failed.
>>
>> Thanks for the links.
>>
>> Many thanks again,
>>
>> Dirk
>>
>>
>> (*)
>> https://dirkgorissen.com/2016/04/19/wheres-susi-airborne-orangutan-tracking-with-python-and-react-js/
>>
>> On 7 February 2017 at 14:22, Marcus Müller <address@hidden>
>> wrote:
>> > Hi Dirk,
>> >
>> > Well, if you have a let's say 2 kHz uncertainty in the frequency of your
>> > pulse, I'd really start very simple. Looking at your plots, I think we
>> > can
>> > sufficiently suppress the noise simply by low-pass filtering:
>> >
>> > * Use a Low Pass filter (real taps) with a cutoff of 1 kHz – that will
>> > let
>> > everything from -1 kHz to +1 kHz around your center frequency through.
>> > You
>> > can use the "freq xlating FIR" block to first shift by a given frequency
>> > and
>> > then apply the filter; might make things easier.
>> > * Detect power by putting your complex samples through a complex mag^2
>> > block.
>> >
>> > If the output of that looks somewhat OK, then you could e.g. low-pass
>> > filter
>> > the result to get rid of noise pulses that are too short, for example :)
>> >
>> > By the way, it might be cool to put up a short recording (ie. a file
>> > generated using the GNU Radio file sink) somewhere (zipped, e.g. on
>> > Google
>> > Drive). I haven't worked with that myself, but maybe putting it on
>> > SigIDwiki[1] might be beneficial for people encountering the wildlife
>> > tracker signal later on.
>> >
>> > I actually think your project is pretty cool – short warning: if this
>> > works
>> > out, I'll try to convince you to write a short article for the GNU Radio
>> > blog!
>> > I'd imagine it being about how you approached the problem and what
>> > you've
>> > found, with emphasis on all the problems you had because of GNU Radio
>> > and
>> > maybe also DSP. We surely could use a few more "oh, GNU Radio is awesome
>> > because it works for everyone" articles (like [2]), but what we far more
>> > severely lack is stories of people that actually made progress, but also
>> > struggled, because they didn't know GR for years before writing a blog
>> > post.
>> > One of the problems that we face (and that was a very prominent theme at
>> > the
>> > Panel at FOSDEM [3]) is that we don't really know how to help people
>> > that
>> > are just approaching GNU Radio, and aren't already familiar with a lot
>> > of
>> > the stuff that we've gotten used to.
>> >
>> > Happy tracking,
>> >
>> > Marcus
>> >
>> > [1] http://www.sigidwiki.com/wiki/Adding_a_Signal_Entry
>> > [2]
>> >
>> > http://gnuradio.org/blog/filtering-time-series-data__elemental-building-blocks/
>> > [3] https://fosdem.org/2017/schedule/event/sdr_panel/ ; don't know –
>> > maybe
>> > we'll have recordings of that.
>> >
>> >
>> > On 02/06/2017 09:39 PM, Dirk Gorissen wrote:
>> >
>> > Thanks Marcus & Martin for the responses.
>> >
>> > To clarify, Im working on a wildlife tracking problem but from the air
>> > (drone). Im purely interested in finding out if the pulse (which gets
>> > transmitted at a fixed interval of 1500ms) occurred or not. If it did, I
>> > know Im within some range of the animal of interest. I have no control
>> > over
>> > the transmitter and no further technical details (unfortunately). I did
>> > take
>> > some screenshots to give you an idea what it looks like:
>> >
>> > https://goo.gl/photos/Y3Ea6kJo1ewrcDYM8  (taken with tx at close range
>> > to
>> > static antenna)
>> >
>> > I expect quite a bit of (fairly uniform) background noise from other
>> > electronics to deal with and there are hardware things that can be
>> > improved.
>> > But for the purposes of this thread I want to ensure Im doing the right
>> > things on the DSP side.
>> >
>> > And yes, Im pretty much a DSP novice but happy to learn. So be gentle :)
>> >
>> >>So, my understanding is that your SDR device first downconverts your
>> >> 150.22
>> >>MHz signal to complex baseband, is that right?
>> >
>> > Yes, so think something like an rtlsdr or airspy.
>> >
>> > About the FFT question, thanks I got a bit confused. I now understand
>> > that
>> > if I want FFT over a longer timeframe I should just take a larger size
>> > and I
>> > can vary the sampling rate (e.g., via decimation) to get the resolution
>> > (Hz
>> > per bin) I require.
>> >
>> > Im now wondering if there is a founded way to pick an optimal FFT size
>> > given
>> > my pulse is only 10ms long. Im guessing it should not be much longer
>> > than
>> > the 10ms equivalent but maybe there is a noise tradeoff (?).
>> >
>> > In any case I agree this is a time based phenomenon so a time domain
>> > approach should likely be the main one.
>> >
>> > So it seems I should at least be trying an FIR filter to do cross
>> > correlation. Not quite clear how I would actually do that in practise
>> > with
>> > the gr block though. How would I provide the second input (the synthetic
>> > pulse)?
>> >
>> > I have come across matched filters before but so far failed to bridge
>> > the
>> > gap from theory to code. This is partly the reason I came to gnuradio.
>> >
>> >>You can implement a limited-length autocorrelation/crosscorrelation
>> >> relatively
>> >>easily; we can talk about how that would look like, but my gut feeling
>> >> is
>> >> that this
>> >>might be something that might not make too much sense in your specific
>> >> use
>> >>case.
>> >
>> > Not too sure myself tbh, but happy to take your lead.
>> >
>> >>it's possible cyclostationary estimation methods might be helpful here.
>> >
>> > So I googled this and poked around a bit on
>> > https://cyclostationary.blog.
>> > Looks really interesting but out of my depth here..
>> >
>> > Thanks for the tip on gr-fosphor.
>> >
>> > Cheers
>> > Dirk
>> >
>> > On 5 February 2017 at 12:06, Marcus Müller <address@hidden>
>> > wrote:
>> >>
>> >> Hi Dirk,
>> >>
>> >> nice to have you around, welcome to GNU Radio! I don't know your level
>> >> of
>> >> DSP knowledge, so please excuse if I either throw too many high-level
>> >> concepts at you or assume you could want to read up on something that
>> >> you
>> >> already know. If something in my reply is unclear, please don't
>> >> hesitate to
>> >> ask for clarification.
>> >>
>> >> So, my understanding is that your SDR device first downconverts your
>> >> 150.22 MHz signal to complex baseband, is that right?
>> >>
>> >> So, what is the objective here? You say you need to /detect/ the pulse,
>> >> but for what purpose? Is it just about detecting the presence of the
>> >> pulse,
>> >> or is it used for eg. time detection?
>> >>
>> >> Your filter->decimate approach sounds very reasonable; I'm not 100%
>> >> convinced by using the FFT for something that is concentrated in *time*
>> >> domain, but that might depend on the purpose mentioned above.
>> >>
>> >> 2) Im somewhat confused about the FFT block if I just pipe the SDR
>> >> straight into it. The FFT size is set to 1024 and the window is set to
>> >> "window.blackmanharris(1024)". So Im assuming the FFT just applies one
>> >> window (?) and outputs 1024 bins. However, how many samples are
>> >> accumulated before the FFT is run? I would have assumed I can control
>> >> that too. And if so, should I best be doing this every 50ms, 500ms,
>> >> 2000ms, ..?
>> >>
>> >> I'm not sure I understand the question. An FFT is simply an DFT. It's a
>> >> mapping of N-dimensional vector to N-dimensional vector, . The window
>> >> is
>> >> multiplied point-wise with the input vector prior to the DFT to avoid
>> >> spectral leakage.
>> >>
>> >> There's no accumulation involved anywhere.
>> >>
>> >> 3) I can use the rational resampling block to bring the sample rate
>> >> down to 48khz so I can use the audio sink. From that I can still hear
>> >> the pulse even if it is not visible in the spectrum (gui sink). Im
>> >> assuming this is just because the plotting cannot keep up?
>> >>
>> >> Maybe, or maybe the spectrum sink really isn't the right tool to
>> >> visualize
>> >> a pulse! Again, we might want to discuss what this pulse is and what
>> >> it's
>> >> used for.
>> >>
>> >> 4) In the time domain I guess I can generate a synthetic pulse of the
>> >> same length / frequency and then cross correlate.
>> >>
>> >> Hm, but correlating a signal with a known fixed sequence is,
>> >> mathematically, a convolution.
>> >>
>> >> That is identical to doing FIR filtering – in fact, if we consider the
>> >> shape of your pulse as what is often referred to as pulse shape filter,
>> >> then
>> >> that filter in the receiver would simply be the matched filter to that.
>> >>
>> >> Not obvious to me
>> >> how to generate the required pulse in gnuradio though (would a
>> >> continuous signal work?).
>> >>
>> >> "Continuous" would mean you'd do an infinite-length correlation, so
>> >> that's
>> >> not 100% possible.
>> >>
>> >> I also notice there are no built in
>> >> (auto)correlation blocks?
>> >>
>> >> Hm, but an autocorrelation would take a complete signal, shift it by
>> >> all
>> >> possible shifts and calculating the dot product between the shifted
>> >> version
>> >> and the unshifted, right? That would require to have the complete
>> >> signal at
>> >> once.
>> >>
>> >> But GNU Radio is a streaming architecture, so that can't work.
>> >>
>> >> You can implement a limited-length autocorrelation/crosscorrelation
>> >> relatively easily; we can talk about how that would look like, but my
>> >> gut
>> >> feeling is that this might be something that might not make too much
>> >> sense
>> >> in your specific use case.
>> >>
>> >> I found the "correlation estimator" but not
>> >> clear how to use it. As for dealing with the frequency uncertainty
>> >> problem. Does one just try correlating with different freuencies and
>> >> pick the best one? Or what is the good thing to do here given I may
>> >> also have to deal with quite a bit of noise.
>> >>
>> >> As a gut feeling: you don't really care about whether the pulse is
>> >> exactly
>> >> at a certain frequency (it's absolutely not normal for wireless
>> >> receivers to
>> >> know the exact frequency a priori), but when it happens. So we might
>> >> want to
>> >> discuss the kind of pulse, and kind of noise we're talking about.
>> >>
>> >> As a further gut feeling: I think your autocorrelation question
>> >> indicates
>> >> you might be on a very good track – it's possible cyclostationary
>> >> estimation
>> >> methods might be helpful here.
>> >>
>> >>
>> >> Best regards,
>> >>
>> >> Marcus
>> >>
>> >>
>> >>
>> >> On 02/04/2017 11:39 PM, Dirk Gorissen wrote:
>> >>
>> >> Fist of all, while Im a newbie to (gnu)radio, congrats to the dev team
>> >> for a great piece of software.
>> >>
>> >> My question is about the need to detect a weak, noisy, short (10ms)
>> >> pulse that occurs every 1.5 seconds. It is transmitted at a particular
>> >> frequency (e.g., 150.22 MHz) but in practise I have found this can
>> >> vary by as much as +/- 500Hz. There is no modulation, it is simply an
>> >> on/off keyed pulse.
>> >>
>> >> Say I have an SDR generating data at 2.5 MSPS. I have so far been
>> >> experimenting with standard scipy/numpy routines to collect a batch of
>> >> samples, perform a FFT (freq domain) and do a cross correlation (time
>> >> domain). However, Im by no means a dsp guy and would like to leverage
>> >> gnuradio as much as I can. I have been poking a bit but have some
>> >> basic questions.
>> >>
>> >> 1) 2.5 Msps gives me way more bandwidth than I neeed. Assuming, for
>> >> now, I only care about a single pulse frequency I really only need
>> >> ~1khz bandwidth. In the frequency domain I can directly decimate down
>> >> (with a big factor) to the 1-2 khz range using the low pass filter
>> >> block, do an fft, and look for peaks. Is that the right approach?
>> >>
>> >> 2) Im somewhat confused about the FFT block if I just pipe the SDR
>> >> straight into it. The FFT size is set to 1024 and the window is set to
>> >> "window.blackmanharris(1024)". So Im assuming the FFT just applies one
>> >> window (?) and outputs 1024 bins. However, how many samples are
>> >> accumulated before the FFT is run? I would have assumed I can control
>> >> that too. And if so, should I best be doing this every 50ms, 500ms,
>> >> 2000ms, ..?
>> >>
>> >> 3) I can use the rational resampling block to bring the sample rate
>> >> down to 48khz so I can use the audio sink. From that I can still hear
>> >> the pulse even if it is not visible in the spectrum (gui sink). Im
>> >> assuming this is just because the plotting cannot keep up?
>> >>
>> >> 4) In the time domain I guess I can generate a synthetic pulse of the
>> >> same length / frequency and then cross correlate. Not obvious to me
>> >> how to generate the required pulse in gnuradio though (would a
>> >> continuous signal work?). I also notice there are no built in
>> >> (auto)correlation blocks? I found the "correlation estimator" but not
>> >> clear how to use it. As for dealing with the frequency uncertainty
>> >> problem. Does one just try correlating with different freuencies and
>> >> pick the best one? Or what is the good thing to do here given I may
>> >> also have to deal with quite a bit of noise.
>> >>
>> >> Any guidance appreciated.
>> >>
>> >> Many thanks,
>> >>
>> >> Dirk
>> >>
>> >> _______________________________________________
>> >> Discuss-gnuradio mailing list
>> >> address@hidden
>> >> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>> >>
>> >> _______________________________________________ Discuss-gnuradio
>> >> mailing
>> >> list address@hidden
>> >> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>> >
>> > --
>> > _________________________________________ Dr. Dirk Gorissen Mob:
>> > +44-7763-806-809 Web: dirkgorissen.com Skype: dirk.gorissen Twitter  :
>> > twitter.com/dirkgor LinkedIn: linkedin.com/in/dirkgorissen
>>
>>
>>
>> --
>> _________________________________________
>> Dr. Dirk Gorissen
>> Mob: +44-7763-806-809
>> Web: dirkgorissen.com
>> Skype: dirk.gorissen
>> Twitter  : twitter.com/dirkgor
>> LinkedIn: linkedin.com/in/dirkgorissen
>>
>> _______________________________________________
>> Discuss-gnuradio mailing list
>> address@hidden
>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>
>



-- 
_________________________________________
Dr. Dirk Gorissen
Mob: +44-7763-806-809
Web: dirkgorissen.com
Skype: dirk.gorissen
Twitter  : twitter.com/dirkgor
LinkedIn: linkedin.com/in/dirkgorissen