Re: [Discuss-gnuradio] RAM consumption

[Eric Blossom]

On Sun, Oct 24, 2010 at 02:28:04PM -0700, David Knox wrote:
> 
> 
> Eric Blossom wrote:
> > 
> > 99% of the base code distributed with GNU Radio (runtime + blocks)
> > allocates NO MEMORY once the flow graph has started (see below for
> > exceptions).
> > 
> > DO NOT use a vector_sink for anything other than QA code that
> > produces a small amount of output.  The vector_sink will allocate an
> > unbounded amount of memory if you keep writing to it.
> > 
> > It is also possible to consume memory by sending an unbounded number
> > of messages into a message queue (gr.msg_queue) that doesn't have a
> > queue size limit and has a slow (or compute bound) reader.
> > 
> > If there's a GUI of any kind involved, or python code that's being
> > executed beyond initialization time, look there for problems.
> > 
> > Eric
> > 
> 
> Thanks for such a quick response.  I am modifying C++ code, rather than
> Python code.  There is no GUI (just an XTERM accessed via std::cout
> statements).  I am using std::queue data structures and the push and pop to
> these queues occurs in the same routine (see below).  

Is there only a single thread that manipulates the queue, or is one
for example inside of a GNU Radio block, and one is outside of the
block?

> I think that your vector sink comments refer to the Python construct, don't
> they?  What is the C++ equivalent of your 'DO NOT DO THIS'
> statement?

They map 1:1 from Python to C++:  

  gr.msg_queue -> gr_msg_queue
  gr.vector_sink_* -> gr_vector_sink_*

> Does this mean I have to access the queue in a critical section or using 
> explicit
> thread-safe methods?

As in any multithreaded programming, if there is more than one thread
involved, and you are accessing a shared data structure from more than
one thread, and somebody else isn't already handling the critical
section for you, you WILL have to implement a critical section.

STL containers are not inherently thread-safe.

> My output seems to be sequenced just fine and there
> are no duplicates or the like.

> My C++ implementation is:
>     .....
>     if ( (int)queue.size() >= MAX) { queue.pop() } 
>     queue.push(value)  
>     .....
> It is possible that the CPU is not able to keep up with the emptying task,
> but then I'd expect the queue size to be growing beyond MAX and it isn't...
> based on std::cout output anyway.  I suppose that this output could be also
> lagging further and further behind, but the output all appears to complete
> properly as each of my relatively rare test events occur (~1 second apart).
> 
> Is there something specific that I must do inside the C++ code to avoid
> GnuRadio run-time memory allocation issues?

No.

> I have not specifically added
> any new malloc/dalloc/calloc statements to the existing code myself.  Could
> declaring data structures (e.g. scalars, arrays, std::queue) inside or
> external to subroutines have this kind of side effect?

Of course they could.  If you're using some kind of container, you
need to know what it's worst case running time and memory usage is.

> Would I be better
> off implementing the queue using an array/circular buffer or std::vector of
> my own, rather than using the one from the std:: library?  

It all depends.

Have you written a new C++ block for GNU Radio?
If so, have you written any QA code for it?

If you've written a block, and you replace the guts of your work or
general_work method with:

  return noutput_items;         // turn block into a NOP

does the memory increase stop?

If you haven't written a new block, how are you getting access to the
samples?  

> My overall objective is that I want to be able to capture and print off
> samples from the ADC on a specific trigger characteristic of the sample
> values (e.g. magnitude).  So, I need to remember (either print out to the
> XTERM or store to a file) small subsets of contiguous ADC samples that
> occurred just prior to my trigger condition.  Once they are printed, I want
> to 'forget' them entirely and start looking for my trigger again.  In my
> testing, this amounts to printing about 50 lines every second or so, but I
> am consuming RAM at a steady rate of about 3 MB/s.

Seems simple enough.  Instead of us trying to guess what you're actually
doing, can you post a link to the code including an example that
exercises it? 

> / David Knox

Eric