Re: [Discuss-gnuradio] Buffer Overflow Debug

[Martin Braun]

Hey Richard,

I'm going to need some time, peace and quiet to get to the bottom of
this. Which means not in the very near future :) But I'm very interested
in getting this to work. Can you post some failing codes/OOTs somewhere
for me to start testing?

M

On 22.05.2015 15:22, Richard Bell wrote:
> Martin et. al.,
> 
> I implemented my own block that produces a trigger signal when a
> specific tag is encountered, and I feed this into the detect port of the
> HPD block. This is an attempt to overcome the tag trigger issues we've
> been having in this email chain. The trigger_tag block simply outputs
> zeros until a specified tag_key is found, at which point it outputs a 1
> on that sample and then continues outputting zeros. It has the same
> trigger behavior as the Schmidt Cox block used to create the detect
> signal in the OFDM example.
> 
> This has resolved the HPD block freezing issue, however, a new issue has
> arisen. Now the block will output the correct header and payload for a
> little while, and then randomly start passing the wrong header portion
> through, even though the detect signal is still on the correct trigger
> point at the input to the HPD block. I confirmed this by comparing the
> HPD port0 input with the HPD port1 input, confirming the detect peeks
> correspond exactly with the port0 tags that signify a header start. Even
> while the HPD block allows the wrong header portion through, the detect
> signal is aligned perfectly with the header start at input0.
> 
> If you wouldn't mind confirming my HPD block settings, I will detail
> them here. I don't think this could be the issue, because I expect the
> block would never work correctly if they were off, but better to be
> safe. The input0 type is complex unpacked data. My header is 32 bits
> long packed. I correspondingly set the following:
> 
> *Header Length (Symbols): 4
> Items per Symbol: 8
> *
> *Guard Interval (items): 0
> *
> *Output Format: Items
> *
> *IO Type: Complex
> *
> *Sampling Rate: <my rate>*
> 
> Now, to add on to this confusing block, I tried using the following
> settings and it worked better than the above settings. However, looking
> at the source code, it makes no sense to me why it should work at all.
> This is because the payload size is calculated by using the
> packet_length read off the message passed in multiplied by the
> items_per_symbol variable. This would correspond to a huge payload
> (128*32 in my case, instead of 128*8), yet it works.
> 
> *Header Length (Symbols): 1
> Items per Symbol: 32*
> 
> Using the other possible combination that produces the correct header
> length when multiplied together, it does not work for any length of time.
> 
> *Header Length (Symbols): 2
> Items per Symbol: 16*
> 
> I'm starting to think there is a major bug in the HPD block when used
> without the 'Output Format Symbols' mode. I say this because this is the
> last difference between the OFDM example, which seems to work fine, and
> my own. At this point I'm stuck code diving into the HPD block to see if
> I can figure out what's going on.
> 
> v/r,
> Rich
> 
> 
> 
> On Wed, May 20, 2015 at 6:27 PM, Richard Bell <address@hidden
> <mailto:address@hidden>> wrote:
> 
>     Yes I will file a bug.
> 
>     We put a tag_debug block right before the HPD input. The tag debug
>     stdout statements continue, while the get_tags_in_range function of
>     the HPD block returns none.
> 
>     We will continue debugging this to see if we can learn something
>     more concrete. The time it takes the flow graph to generate this
>     error is very variable. From hours to seconds.
> 
>     If there is other useful debug info you'd like us to generate please
>     let me know and we will.
> 
>     V/r,
>     Rich
> 
>     > On May 20, 2015, at 6:05 PM, Martin Braun <address@hidden
>     <mailto:address@hidden>> wrote:
>     >
>     > This is interesting, and kinda serious. Also, we've had reports that
>     > tags go missing in the past, but it's something that's hard to verify.
>     >
>     > How did you confirm the input stream is correctly tagged? If
>     > get_tags_in_range isn't finding tags that it should, that is most
>     likely
>     > the issue you're seeing here.
>     >
>     > Can you please file a bug report on the issue tracker, with as much
>     > detail as you can provide?
>     >
>     > On a side note, you might be able to use the trigger input instead of
>     > the tags, but that doesn't solve this problem.
>     >
>     > Thanks,
>     > Martin
>     >
>     >> On 20.05.2015 11:02, Richard Bell wrote:
>     >> Hi Martin,
>     >>
>     >> Sorry for the delay in response. We have been able to put some
>     time into
>     >> debugging this issue and here is what we've found:
>     >>
>     >> 1) We have confirmed that the input stream to the HPD block is
>     correctly
>     >> tagged when the block freezes. The tags we set as "trigger tag"
>     in the
>     >> block are on the input stream.
>     >>
>     >> 2) At some point in the flowgraph operation, the HPD block gets
>     stuck in
>     >> the STATE_FIND_TRIGGER state (idle state). This is even though the
>     >> trigger tags are present on the input, as confirmed in 1 above.
>     We are
>     >> observing that get_tags_in_range is failing to find the tags in the
>     >> stream. We can't figure out what would cause this, or that this
>     is even
>     >> the issue. It's the best idea we have.
>     >>
>     >> So we agree there is no buffering issue. The issue is with the
>     HPD block
>     >> not seeing tags that are confirmed to enter the input port. We would
>     >> love some help on this issue.
>     >>
>     >> The way we are debugging, is we copy and pasted the built in HPD
>     block
>     >> source to our own custom gr_modtool module and added cout debug
>     >> statements there.
>     >>
>     >> v/r,
>     >> Rich
>     >>
>     >> On Thu, May 14, 2015 at 4:05 PM, Martin Braun
>     <address@hidden <mailto:address@hidden>
>     >> <mailto:address@hidden <mailto:address@hidden>>>
>     wrote:
>     >>
>     >>>    On 14.05.2015 15:26, Richard Bell wrote:
>     >>> Hi all,
>     >>>
>     >>> I'm working on an incredibly annoying issue related to my use of the
>     >>> Header/Payload Demux (HPD) block. I think it's related to a buffer
>     >>> overflow at some point, but I'm having a really hard time coming
>     >>    up with
>     >>> a proper debug strategy to nail this down.
>     >>>
>     >>> What I'm seeing is my data streams freeze after the input to the HPD
>     >>> block, both on the header branch and the payload branch. Everything
>     >>> before the HPD block continues on without issue. The time it
>     takes the
>     >>> streams to freeze is HIGHLY variable. I've watched it run for 30
>     >>    minutes
>     >>> straight before a freeze and I've watched it freeze a few seconds
>     >>    after
>     >>> start. I'm using tags generated by the Correlation Estimator as the
>     >>> trigger for the HPD block.
>     >>>
>     >>> My question is this, if I suspect a buffer overflow is causing a
>     >>    freeze,
>     >>> how would I prove this to myself?
>     >>
>     >>    Rich,
>     >>
>     >>    a "buffer overflow" wouldn't cause GR to freeze, rather, it
>     would crash.
>     >>    Going by your previous messages, I suspect what you're seeing
>     is that
>     >>    the HPD is starting to block, causing backpressure until that
>     in turn
>     >>    reaches the source. (Correct me if I'm wrong).
>     >>
>     >>    I remember you previously mentioning something similar. Did
>     you confirm
>     >>    the header parser is actually sending out a message for every data
>     >>    packet it receives? This is a case where the HPD is actually
>     designed in
>     >>    a way that it'll fail.
>     >>
>     >>    As a debugging strategy, I would recommend printing out the state
>     >>    changes inside the HPD state machine. If it freezes, it would be
>     >>    interesting to see in which state that is.
>     >>
>     >>    Cheers,
>     >>    Martin
>     >>
>     >>
>     >>    _______________________________________________
>     >>    Discuss-gnuradio mailing list
>     >>    address@hidden <mailto:address@hidden>
>     <mailto:address@hidden <mailto:address@hidden>>
>     >>    https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>     >
> 
>