Re: [Discuss-gnuradio] Dropping samples "D" when CPU at 60%

Hi Marcus,

Thanks for looking into the block. You're right that it's a block from trial and error, and the moving average block is what I wanted.

I don't think that I'll observe a long fade effect in my setup (indoor + <10MSps sampling rate + mostly line of sight. Most multi-path components should arrive in the same sample as the direct path's).

Coming from a non-signal processing background, tend not to think of things in the frequency domain, and so I stuck to time. This may come back to bite me later, but that's future me's problem.

I've attached a plot (from simulation) for the output at almost each step in my block.

1--- The divider(blue) simply tells me when the signal(orange) power has increases more than X times w.r.t. older samples. It isn't exact, but good enough.

2--- However, the divider output is unbounded, so I threshold it (green).

3--- The threshold block's output holds at 1 until the input drops below a value, and doesn't give an impulse(green).

4--- So I do the delay and subtract so that only transition points have non-zero values(red).

5--- Of which I want the 1st transition only, which is why I threshold again.

It isn't exact, but gets the job done.

Thinking about it now, I think that I should offload some/most of these steps (steps 3-5) to another block that doesn't operation on all incoming samples.

Thank you,

Regarding your flow graphs:

energy_det:
###########

Moving average
==============

Your Decimating FIR is really just a moving average. Try that block; it
uses far less operations per sample.

Delay-Divide
============

This is a very nonlinear operation:

---------------->Divide
-->Delay160----^

It's *probably* something like an f_sample/160 notch filter with
terrible, terrible spectral properties?

Are you *sure* this is what you want? Can you give a motivation for
this?

To motivate my doubts:

You get a peak at the output of your divider all 160 samples after a
very low power instant, 1/(x[t-160]). The actual energy at that point
plays a minor role, the fading depth is what makes the peak. Because
your filter only has a "memory" of 50 samples, what happens 160 samples
later is only linked through the channel's autocorrelation function.

I recommend looking at plots of typical channel realizations[2] to see
why that's a bad idea.

You'd need very low doppler spreads (in relation to your sampling
rater) for e.g. a Rayleigh model to make it likely enough that your
channel 160 samples (I approximate that in my head with a Gaussian with
variance ca doppler spread †) later is still correlated enough to just
avoid arbitrarily high peaks that really don't lend themselves to power
detection.

Division is a tool *very* rarely used with powers in signal processing,
exactly for its stability problems.

Delay-Subtract
==============

Your

-> Delay->Subtract
----------^

is really just a [1,-1] FIR Filter. Use that, and save some processing power. It's a half-band high pass filter, see the plot in [1].

Or don't -
because the "threshold" before that will inherently give you an impulse that would pass the filter.

Threshold--High Pass--same Threshold
=====================================

So, the filter probably doesn't do anything useful, since you've definitely killed all high-frequency components with the moving average of length 64.
The thing here is that at frequencies where your delay-subtract passes signal,
you only see the (slowly decaying) side lobes of the sinc that is the frequency response of your moving average FIR.
And these side lobes have the unfortunate property of being f_sample/64 wide and having nulls in between!

So, what you're seeing at the output of that is totally dependent on *miniature* fluctuations of the periodicity of changes in receive power.
This makes little sense for an energy detector.

The second threshold is more than questionable. The rising output edge of the first threshold should inherently pass the second.

Overall
=======

This doesn't look like you've /designed/ it to fit a purpose; more like the result of a trial-and-error session.
Maybe sit down once more and think very closely about what the mechanism should be, and then implement that.

Best regards,
Marcus

†: frankly, please don't quote me on that one. I just looked really hard at the characteristic function of the Rayleigh distribution for 𝜎=1.8 and said "naaah" aloud.

[1]
[2] https://en.wikipedia.org/wiki/Rayleigh_fading

On Tue, 2018-03-27 at 18:00 +0000, Bakshi, Arjun wrote:
> Hi Marcus,
>
> Thanks. I think it was just that ~18MSps is too much for my PC. After writing a cpp block for detecting the trigger and piping some samples, the cpu utilization jumped up to +90% on all cores, and I can now support 6.25MSps x 3 streams. But not much more. For some reason I expected that my cpu could handle the full 1Gbps connection.
>
> The CPU is an older (Sandy Bridge/gen-2) address@hidden 4 cores, running on Ubuntu 16. Memory wasn't an issue, as the memory usage for the flow hovers around 100MB, with ~10GB to spare.
>
> The ethernet connection is able support higher sampling rates(8Msps x 3 streams) with direct USRP source to null sink connections. So the ethernet/network should be fine.
>
> I've attached screenshots of my flowgraphs, and some code for the blocks I use.
>
> Only concern now is if its harmful to have 90%-100% cpu utilization for an hour or so.
>
> Thanks,
>
> AB
> From: Müller, Marcus (CEL) <address@hidden>
> Sent: Tuesday, March 27, 2018 4:39:10 AM
> To: address@hidden; Bakshi, Arjun
> Subject: Re: [Discuss-gnuradio] Dropping samples "D" when CPU at 60%
>
> Dear Bakshi,
> again, we'd need to get a holistic view of what you're doing. All we
> can do is speculate wildly based on your description.
>
> > Is it just that the block I've written is not efficient enough? I
> would expect higher CPU utilization in that case.
>
> Possibly. Don't know. You tell us nothing about your block, especially
> no code.
>
> Please be more precise when asking, give your code, give actual
> throughput benchmark, give us hardware information (which I've asked of
> you before, but you never delivered). It's actually not easy to ask a
> good question, but you really need to get better at it!
>
> Fair warning: a sum rate of 18.75MS/s is really not that little. It's
> absolutely possible you have shoddy network hardware or drivers that
> just can't deal with that much data. Again, pure speculation, since you
> tell us nothing about your hardware, or your OS, or anything about the
> environment you're working in.
>
> All that we can say is: something in your overall system, be it
> hardware, your CPU, your RAM, your software or anything else, doesn't
> work as you want it to.
>
> Best regards,
> Marcus
>
> On Sat, 2018-03-24 at 19:30 +0000, Bakshi, Arjun wrote:
> > Hi all,
> >
> > I have written a block in python to detect and decode some packet transmissions. I've noticed that when I have 3 rx streams running at 6.25Msps (total network traffic is 75MB/s), the host starts dropping samples after a while. The CPU utilization does not exceed 60%, and the Ethernet isn't being saturated. I've increased buffer sizes (net.core.w/rmem_max) to the largest values possible
> >
> > A 'D' in the console means that the host isn't consuming samples fast enough.
> >
> > Is it just that the block I've written is not efficient enough? I would expect higher CPU utilization in that case.
> >
> > I think the most expensive thing I'm doing is looking at each sample in a stream of floats to find where the values goes above a threshold. This part is in my block, and triggers the decoding logic, which is in the same block.
> >
> > I don't know how to do this more efficiently. I'm thinking of re-writing it in C++, but I don't know how much better it'll get.
> >
> > Thank you,
> >
> > AB
> > _______________________________________________
> > Discuss-gnuradio mailing list
> > address@hidden
> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
> Discuss-gnuradio Info Page - lists.gnu.org lists.gnu.org
> Discuss-gnuradio -- GNU Radio, the Free & Open-Source Toolkit for Software Radio About Discuss-gnuradio

From:	Bakshi, Arjun
Subject:	Re: [Discuss-gnuradio] Dropping samples "D" when CPU at 60%
Date:	Wed, 28 Mar 2018 21:37:43 +0000