Re: [Discuss-gnuradio] inefficient large vectors

[Marcus Müller]

Just to add my two cents:
Depending on your actual application, your large vectors might not actually quite fit the 
idea of "streams"; they might, for example, be a valid, decoded network packet 
or something of the like. If they don't need sample-synchronous handling, using messages 
to pass them around might work well.
Downside of that is of course that you can't use your favourite GR block on 
messages. You break the sample synchronous architecture of a flowgraph with 
multiple paths from source(s) to sink(s); and if you convert from message to 
stream and back, you basically lose the vector attribute of your data (or run 
into the same problems as before).
I can't really tell you much about computational performance of passing around 
large messages, however.

On the other hand, you can reduce your per-block coding overhead for Martin's 
suggested tag-based solution:
Write a base class that implements a input and output buffer and a minimal 
state machine based on stream tag evaluation. Let your blocks inherit from 
that. Always copy as many items from your general_work's input vector to your 
input buffer as you can, and copy as many samples from the output buffer to 
your general_works output vector as possible. Execute your computation when 
your input buffer is full and your output buffer empty. That way, you'll get a 
quasi-fixed relative rate, but get all the freedom and scheduling disadvantages 
of a general_work block with an itemsize of gr_complex (or whatever your data 
type is).

I know from experience that this might be hard to debug. However, once your 
state machine is watertight, you're not very likely to run into issues later.

Happy hacking,
Marcus

On 08/21/2013 07:59 PM, Miklos Maroti wrote:
> Hi!
>
> I have many sync blocks that work with large fixed size vectors, e.g.
> converts one vector of size 12659 to another with size 18353. I have
> just multiplied the sizeof(gr_complex) with 12659 and 18353 in the
> signature. However, when the flow graph is running, then I get a
> warning about paging: the circular buffer implementation allocates
> large buffers (e.g. 4096 many to make the paging requirement). I do
> not want really large buffers. I have implemented the whole thing with
> padding, but that becomes also really inefficient, since when you want
> to switch between vectors and streams, then you have to jump through
> extra hoops with the padding. In a previous version I had streams
> everywhere, but then there is absolutely no verification whether I
> messed up the sizes of my "virtual vectors".
>
> So is there a way to work with large odd length vectors which does not
> have this buffer allocation problem, and does not require padding? It
> seems to me that it could be supported: regular streams but the vector
> size would be verified separately at connection time and would not be
> used to multiply the item size. Any advice is appreciated...
>
> Best,
> Miklos
>
> _______________________________________________
> Discuss-gnuradio mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio