Re: [Discuss-gnuradio] dealing with framed data

[Kevin Reid]

On Apr 23, 2014, at 8:05, Matthieu Imbert <address@hidden> wrote:

> I'm currently trying to take the output of a fft block and pipe it in a tcp 
> sink for (near) real-time processing outside of gnuradio.
> 
> My question is:
> 
> Assuming that I have vectors of n floats (the fft block outputs vectors of 
> floats or complexes, so I may need to pipe it to a complex-to-mag2 block to 
> convert it to an array of floats, but that's not the main point of my 
> question here), if I send them directly to the tcp sink, it will become a 
> stream of bytes, and my code outside of gnuradio won't be able to detect the 
> boundaries between vectors of n bytes. Actually it won't even be able to 
> detect the boundaries between floats.
> 
> So I need to "frame" the data in a kind of basic protocol. For example: each 
> vector of n floats, insert a m bytes header with a magic identifier, and 
> perhaps a length and/or sequence number.

Actually, you don't need any framing, and the boundaries can be well-defined.

Unlike a radio transmission, TCP is a reliable protocol -- it delivers the 
bytes that are sent without any omissions. Furthermore, there is a defined 
start-of-data point (when the connection was created). Therefore, you can 
simply _count bytes_.

The first 4 bytes the receiving program gets make up a float. The next 4 bytes 
are another float, and so on. Collect groups of 4 bytes to make floats, then 
collect groups of N floats to make arrays of floats.

The one tricky bit is that (at least in POSIX) there is no guarantee the socket 
read operations on the receiving side will give you multiples of 4 bytes, so 
you will need to buffer at least up to 3 bytes (or an entire vector length), or 
use high-level facilities in your language/framework to read guaranteed lengths.

The only reason to use a more elaborate protocol (with headers or whatever) on 
top of TCP is if your data is not homogeneous -- if it consists of 
variable-length messages. That is not the case here (unless, for example, you 
want to add the capability to change your FFT-length mid-stream).

-- 
Kevin Reid                                  <http://switchb.org/kpreid/>