Re: [Discuss-gnuradio] Recent development activity, upcoming plans

[Tom Rondeau]

On Mon, Apr 2, 2012 at 2:17 AM, Josh Blum <address@hidden> wrote:
>
>> - Removal of dependency on gsl for gnuradio-core. The wavelet blocks
>> that were in gnuradio-core, which depend on the GNU Scientific Library
>
> I would make a similar argument for the FFTW dependency on
> gnuradio-core. I'm not volunteering, but that would bring gnuradio-core
> down to gruel, boost, and cppunit as dependencies. Even cppunit could be
> replaced with boost unit test.

FFTs are pretty vital to signal processing and are used in multiple
places. Many of these are in the filters, and we have a plan (for 3.7)
to create a gr-filter category. You aren't going to get a whole lot
accomplished in signal processing if you don't have filters and the
components that use FFTW. So even if we made a separate component for
FFTs, it's still and _effective_ requirement to install it even if
it's not a _required_ requirement.

When we get gr-filter, then we could have a discussion of whether its
appropriate to have a gr-fft component. I feel that's a bit too much
granularity, but we can discuss it at that point. I think as we are
moving towards more top-level components and less in gnuradio-core
(mainly just runtime), we will be working on these issues in general.
We don't want too much granularity, and I still see gr_fft_vcc being a
part of a larger component structure and not just on its own.

Wavelets are their own class of algorithms that are really useful for
lots of things, but aren't fundamental to most processing we do. So on
one level, separating it out means most people won't notice that it's
missing, and on another, there are lots of new things that can be
added that fall under the gr-wavelet component in the future. That's
not really true for FFTs. I had suggested we have a gr-scientific or
maybe a gr-transforms, but I don't think that plays too well, really,
even if it is technically appropriate (and if we did FFTs and wavelets
in one component, we'd force people to install GSL just to get FFTs;
not where we want to be going with this).

Basically, I'm saying this is a discussion for a little while in the
future. Definitely at least after we've move gr-filter out of
gnuradio-core.

As for the unit testing, frankly, I don't care what tool is used so
long as the behavior is the same as we have now. One thing that's
important is that the QA results are stored in the XML files that
Jenkins reads. I'm pretty sure Jenkins can handle Boost output files,
too, but we'd want to make sure that was working cleanly if we made
the move. What I don't want is to have two such systems running (we
already have a Python and C++ unit test, which some find confusing
already). So an switch would have to be made all at once. I personally
don't see that it's such a big deal to go through that effort to make
the change.

>> - Preparation to to merge the 'next' branch back in to 'master', after
>> the 3.5.3 release.  The 'next' branch is the long-running development
>> branch where we implement things that are either somewhat unstable, or
>> more often, make changes to the API such that user code might break.
>
> There was this idea of loadable modules for gr-audio. That way one could
> build gr-audio for all the audio development environments, but when
> deploying, you only install the modules that the OS has runtime support
> for. Think deb packaging, where each module becomes a package with
> specific requirements like libjack, libalsa... Would this merge event be
> a good time to implement something like that?
>
> Also, for those of us who are interested in gnuradio's PMT library:
>
> 1) Would this be a good time to fix the tag/PMT mutability issue? This
> would prevent multiple downstream blocks from being able to manipulate a
> shared resource, but it is a slight API change for the tags:
> http://gnuradio.org/redmine/issues/490

Johnathan and I have discussed this, but I don't think we've come up
with a plan. The idea is that we fix the issue of being able to
manipulate pmt_vectors (and any other containers) after their creation
such that all PMTs are now read-only. The PMT vectors are a good idea
in that you want to be able to have a vector of items, but the
implementation has allowed them to be changed, which produces thread
safety issues.

> 2) I would love to see the concept of tag/PMT based message passing make
> it into the scheduler. The basic idea is to pass tags (without samples
> associated) between blocks as a form of out-of-band
> communication/control-plane. Here is here squashed into a single
> changeset: https://github.com/guruofquality/gnuradio/tree/msg_passing

I'm of the mind that yes, this is a good thing. The squashed commit it
nice, so I finally went through to look at the code. This can probably
be merged in soon. Johnathan and I are going to take a closer look at
it, so we might have a few requests for it soon.

One request. Could you write either some documentation for how to use
the message passing system or, probably better, a simple example that
shows connecting blocks together and passing messages between them?
That would help clarify their use. I'm not saying that this is a
requirement before the merge, but it would be nice to see how it's
used.

> 3) I would also like to purpose some extensions to the PMT library to
> make PMTs more intelligent about memory management and allocation
> overhead. Also, the work is squashed into a single changeset:
> https://github.com/guruofquality/gnuradio/tree/pmt_extensions
>
> -josh

There are a couple of problems with the work here. First, there is the
issue of adding a read/write buffer to the pmt_blobs. That produces
thread safety issues with PMTs. The idea of a PMT is that if you need
to change the contents, you create a new PMT with the new contents.
Yes, this generates overhead in creating the PMT and doing any memory
copying, but we don't know that this is actually a problem right now.
Also, while this might cause some overhead to be incurred when working
with PMTs, the alternative is to introduce the possibility of
segfaults. PMTs are meant to be used in different threads and even
processes. It becomes a nightmare to think about mutex locking them
all over the place for concurrency. Having them read-only keeps us
from having to worry about this. I think the current PMTs that can be
manipulated after their creation are bugs that we need to fix. Not the
other way around.

This unfortunately makes the PMT pool idea unworkable. If I understand
it correctly, the idea is to create a set of PMTs that can then be
grabbed and modified instead of always creating new ones. This can't
work since PMTs are immutable. Unless you are doing it some other way
that doesn't require this. But again, I'm not sure where PMT creation
is the bottleneck anywhere, and Boost intrusive pointers work really
fast already.

I hope this clears up why the PMT read/write issue is a non-starter.
I'd rather see a bit of slowdown then segfaults :)

Tom (with input from Johnathan)