Re: [Discuss-gnuradio] Memory allocation woes

[Eric Blossom]

On Mon, Jun 28, 2010 at 09:14:34PM -0400, Marcus D. Leech wrote:
> Spent some time tracking down a memory allocation issue.  The SYSV shm
> allocator was getting errors on
>   a request for 1.56GB.  Now, it turns out that the segment size used by
> SYSV shm uses a signed 32-bit
>   int for the size of the segment, which means you can't allocate
> segments larger than 2*31 bytes.  But
>   why was the request for 1.56GB being blown away?  Because the SYSV shm
> allocator in Gnu Radio
>   multiplies the request size by 2 before asking the system for that
> much shared memory.
> 
> So, why does it do that?  And why is the memory allocation so incredibly
> piggish?  We went through this
>   question a couple of years ago, and I'm running into similar problems
> again--my application uses
>   *HUGE* FFTs--1Hz resolution at up to 16MHz (USRP1) or 25MHz (USRP2)
> bandwidth.  This, not surprisingly,
>   leads to some large memory requirements, but Gnu Radios allocator
> seems to allocate a significant amount
>   more than is really needed.
> 
> In the case cited above, the FFT size was 6M bins, which granted is
> outside the "usual" range of most
>   Gnu Radio applications, but I was able to make this work last year up
> to 16M bins.
> 
> The flowgraph involved is quite simple.  A source, a short FFT-based
> filter, the main HUGE FFT, and then
>   a complex-to-mag**2, then a file sink.
> 
> Should this really require gigabytes of memory?

Maybe.

The actual amount allocated depends on a bunch of factors, including
the characteristics of the blocks upstream and downstream from the
buffer in question.

In general, there's a factor of 2x to allow double buffering.  When
using any of the vmcircbuf allocators, there is a temporary request
for twice the desired memory.  This is to ensure that we get a big
enough hole to map what we really want (1/2 that much) into the VM
space twice, contiguously.  This is to allow us to implement the
circular buffer by mapping the same memory into the VM twice, avoiding
the need to special case the boundary condition.

Do you know if it's the buffer upstream or downstream from the FFT
that's failing to be allocated?  Is the block downstream from the FFT
a decimator?  In that case, we allocate N (where N is the decimation
factor) * <the-other-number-we-would-have-chosen> to ensure that
there's enough input for the downstream block to generate a single
output on one call to work.

In general that's not a problem, but in your case, this could be the
part that's causing the large allocation.  It could be worked around
by writing a version of the decimator that doesn't require the factor
of N on its input.  This requires more complicated state tracking in
the block (which is why we don't normally do it), but is possible.


The place to add some debugging outout would be
gr_flat_flowgraph.cc::allocate_buffer.  Something like:

  std::cout << "allocate_buffer " << block
            << " nitems " << nitems << " item_size " << item_size << std::endl;


For the case of the 6M bin FFT, I could see us getting to

  6M * sizeof(gr_complex) * 2 * 2 --> 192MB.  Not that big.


Add the debugging output and let us know what you find.


Also, you could try using an alternate vmcircbuf allocator,
gr_vmcircbuf_mmap_shm, by writing the string 
"gr_vmcircbuf_mmap_shm_open_factory"
into ~/.gnuradio/prefs/gr_vmcircbuf_default_factory.  Note that there
is no trailing newline in that file.  The last byte of the file
should by "y".

gr_vmcircbuf_mmap_shm_open uses an off_t which is 64-bits.
Details in gr_vmcircbuf_mmap_shm_open.cc


Eric