Wow, Greg,
great insight! I've never actually played around with software
like Hadoop, but always planned to (on a small scale – the number
of nodes in my very heterogeneous "home/office cluster" would be
an insane 5).
I really think we should be harnessing the power of the big data
tools in SDR quite a bit more. I totally agree, a lot of SDR folks
definitely have a lot of data to deal with in the first place, and
they might really want to distributedly filter that down to a data
stream that is actually worth storing – in most applications, you
don't care about the original IQ data once you've extracted the
features you wanted – be it the presence of e.g. aircraft
transponders, interference, radar-detected meteors or ocean waves,
atmosphere data or whatever purpose you're doing SDR for in the
first place.
I have always though HDFS + Gnuradio
are destined for each other. It may be a bit early for this
with today's hardware; Mr. Moore is helping us along just fine,
so is AWS.
Hm, do you *think* there's reasonably easy and yet interesting
applications that a student could implement during a Summer of Code?
As far as I understand, HDFS is more or less the storage subsystem
for Hadoop; maybe that's not the only thing worth considering in the
context of SDR here. Is there an interesting application of the
MapReduce paradigm that you'd love to see combined with GNU Radio?
Best regards,
Marcus
On 01/21/2017 05:33 PM, Gregory
Ratcliff wrote:
I spend my working hours on big data and Hadoop.
It occurs to me you really need to be
thinking about something outside of a normal file system. HDFS
lets you write out data in chunks that you later combine when
you have time. There are some really (really) fast
implementation projects that write to hdfs. Most of the new
work is in java, but I think you are asking for something pretty
light.
I can visualize a "gatherer" for RF
and a "filer" in HDFS that writes out xx MB chunks every period.
Now as others have said, you don't just slap some stuff
together, you will need to optimize the integration points and
think about the best caching and write speeds of the "filer"
system and the persistent storage.
Likewise, there are plenty of apache
tools that will recombine the HDFS chunks back into files of
arbitrary size.....which you can then analyze later with
gnuradio...when time doesn't matter as much. You might not need
much of Hadoop that the file system and some tools.
I have always though HDFS + Gnuradio
are destined for each other. It may be a bit early for this
with today's hardware; Mr. Moore is helping us along just fine,
so is AWS.
Greg
Nz8r
I can assure you that 32 GHz is not your sampling rate. Do
you mean 32 MHz?
The problem here is that at first, your operating system
can be smart and cache write accesses to files on mass
storage devices in RAM (or you use a RAM disk, so everything
happens in RAM). But at some point, RAM is going to run out
– and then, your recording speed is effectively limited by
how fast you can write to your storage (in case of a RAM
disk, you simply run full, or your OS starts "swapping", ie.
writing RAM to storage. same problem).
So, unless you find a way to *reduce* the amount of data
you want to record, or simply buy a faster storage system,
there's not much you can do.
Best regards,
Marcus
On 01/20/2017 08:42 PM, Mallesham
Dasari wrote:
Hi Marcus,
Thanks for the quick response. I am recording the FFT
samples continuously. But, I am getting overflow after
some time when the file size has become huge. My sample
rate is high (32GHz) and hence writing to the file takes
so long and hence the usrp_spectrum_sense getting
overflow.
_______________________________________________
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
|