Re: [Discuss-gnuradio] [GSOC17 - Draft proposal] Implement SigMF functionality for GNU Radio

[Sebastian Müller]

Hi Kostis,

I wanted to add some of my thoughts to this. Hope I haven’t missed an answer so far in this extensive discussion.

- Concerning the filtering of datafiles, I think there should be 2 ways to do this. The first one selecting a core / capture / annotation object from the SigMF file which results in a selection/zoom in the time/frequency plots and display of the corresponding metadata in GUI component C. The second could be a „free selection“ in these plots, that again creates the corresponding metadata and enables adding fields (like comment for an annotation object) and saving them as new object to the metadata file.

- I think a standalone tool for converting datasets to SigMF would be sensible (e.g. put that in the „apps“ folder). Since we don’t have a continuous stream of samples here, but a one-time action that requires some user input. You could use the mentioned GUI for that.

- You mentioned filtering for modulation. That is a killer feature, but how are you planning to do that?

- Concerning the GUI I have some eyecandy in mind. Like highlighting capture / annotations objects when displaying the whole time signal. Clicking on one object could enable the mentioned filter for it and „zoom“ into the segment while changing the metadata as well.

- I think you should be more detailed on your timeline.

Best,

Sebastian Müller

address@hidden

Am 25. März 2017 um 01:21:03, Kostis Triantafyllakis (address@hidden) schrieb:

Bastian, Ben, Seth, Nathan and Paul,

Thank you all for your useful feedback!
It's really exciting the fact that this project is a chance for such an interesting discussion.
I will try to make some initial comments on your responses and I'll start working on the updated proposal as soon as possible.

On 03/24/2017 06:01 PM, Ben Hilburn wrote:

Hi Kostis -

Great proposal! I'm really excited to see someone tackling the SigMF item on the Ideas List.

I'll respond in-line to Bastian, and then add some of my own:

- You mention that sources and sinks will handle PDUs and Tagged
Streams. I wonder what would be one PDU. A SigMF "capture segment" or
some other chunk of samples. Also are you sure you want to use Tagged
Streams? Maybe you meant normal streams with tags? With Tagged Streams,
you would have to fit the whole capture segment in the input buffer of
the block. That might not be possible.

I was actually referring to normal streams with tags. I should fix that in my proposal.
Furthermore, I  will examine the case of Tagged Streams in order to be able to safely reject them.

- You mention that RapidJSON code is messy. Does it auto-generate code?
That could even be an advantage, since SigMF might well go through some
more iterations. Did you plan to also create wrappers for Python?

Messy was a word I've chosen to describe that the produced code is not so easy to read and understand.
Python wrappers were not in my initial intentions, but as long as the timeline permits it I would be interested on implementing them.

- The goal of SigMF seems to be to provide a very minimal spec, and I
think many users will have to rely on custom extension in namespaces.
Maybe you could describe a bit how source/sinks and the GUI could deal
with these extensions. 

- You will provide the means to filter data. Did you only plan to filter
specific capture segments or also arbitrary samples based on
annotations? Since you mentioned modulation, which is, AFAIK, currently
not specified, did you plan to allow filtering on custom properties?

So, thinking about this a bit, if Kostis' implementation enables filtering on fields in the `core` namespace, it shouldn't really be any additional effort to support non-`core` namespaced fields. Since any field would be in a capture / annotation segment, which thus would be mapped to a sample index / number of samples, Kostis' software doesn't have to care *what* that field is in order to "select it" in the tool.

That's how I read the proposal, anyway. Kostis, it might be good to clarify this in the proposal.

 

I think Ben's answer covered the concept I had in my mind. For sure I have to rephrase/clarify this in the proposal.

- Regarding the GUI, it seems as if you plan to develop something from
scratch. Did you consider extending projects like "such samples" from
Tim or Inspectrum?

(Seth just responded while I was writing this message, but my comments are similar.)

I agree with Seth and Bastian, here. Generally, I'm a fan of extending existing functionality rather than creating something from scratch. That said, using one of the existing things will also require development.

In the "such samples" case, it's actually just itself a flowgraph (https://github.com/osh/gr-pyqt/blob/master/apps/such_samples2.grc). In order to integrate SigMF into it (which I don't think is a bad idea), new blocks / code will need to be written. Using "Inspectrum" is actually more problematic, especially since it doesn't use GNU Radio under-the-hood. You would effectively be doing work for a different project on GNU Radio GSoC money, which I don't think makes sense.

It might be worth looking at what blocks / code would be necessary to allow the existing Qt GUI features (e.g., "such samples") to make use of the functionality you will be writing in GNU Radio. Ideally, any work you do there can be used in other flowgraphs (not just "such samples"), but using "such samples" would be a great way to reduce your own burden. If you determine that there is really no way to build on things that already exist, then implementing a new tool (like the one in your proposal, now) that is integrated with GRC (e.g., the filter design tool) might indeed be the best way forward. It would be good to explain the justification behind the final decision in the proposal.

(It's worth noting that I would really like to see SigMF support in Inspectrum, and I don't think SigMF support there means it wouldn't be useful here, as well. Inspectrum does more than just visualize data, and one of the key goals of SigMF is portability between tools & workflows.)

 

It is true that in my mind I was thinking the proposed SigMF GUI app as an integrated GRC tool, just like filter design. Nevertheless, I totally understand the philosophy behind reusing/extending instead of re-inventing the wheel. I should give a careful look on gr-pyqt and QT sinks though.

- I wonder what would be the workflow when creating a SigMF recording.
An issue might be that you cannot immediately write the SigMF meta file
during recording. (For example, the hash has to be calculated after the
recording. Or the annotations have to be written after the capture
segments.). Did you plan to create the meta data in memory and write the
file when the flow graph shuts down (which might cause data loss when
you experience a segfault) or do you write meta data to disk in an
intermediate format that you have to post-process manually to create the
final SigMF files.

So the SigMF spec is designed specifically to enable writing the metadata and dataset files in a streaming manner. The spec is careful not to dictate how applications create SigMF recordings, so it's not a problem if you write a bunch of stuff and then after you finish streaming to the files you open them up and add some more stuff to finish them off.

Bastian's point, though, is a good one. In the use-case where you are streaming metadata and samples into files and then you close your application, do you plan on needing to go back to add more information to the metadata file? If so, how do you do that in an automatic way if the flowgraph has stopped?

Maybe I could let the destructor of the block in combination with fseekm handle the finalization of the metafile. The case of segfault is actually a question that needs more attention.

- We are working hard to get the SigMF spec into a stable state as soon as possible, but as you can see from the issue tracker (which is where we are hosting our discussions) there are aspects of the spec that are still being debated. Many of these would impact your implementation. I think it would be good to explain your plan to (a) participate in the SigMF spec discussions, especially with the insight you gain from implementing support for it and (b) deal with pieces of the spec that may be a moving target.

I have already started looking at the open issues. Do you think it would be meaningful to move the discussion there, in a dedicated issue?

- Since this will create a dependency on RapidJSON (or whatever you use), we'll want to be sure that *not* having that installed doesn't prevent building anything not related to SigMF within GNU Radio. I really dislike adding dependencies, but since we don't already have a JSON library there isn't much of an option, here, so we just need to be sure to minimize the burden to users, developers, and packagers.

According to the description in the main page, RapidJSON "is self-contained and header-only". I dont know if MIT is a problem though.

- "to-SigMF Converters": An important tool will be something that converts a file written with the current GNU Radio File Sink into a SigMF recording. I would limit the scope of the converter work to this.

Cheers,

Ben

On 03/23/2017 07:37 PM, Kostis Triantafyllakis wrote:
> Hello,
>
> My name is Kostis Triantafyllakis, graduate student of Computer Science
> at the University of Crete.
> I'm interested in the project of SigMF functionality implementation for
> GNU Radio.
>
> I have started working on my proposal and I kindly ask for your feedback.
> Please find the draft below:
>
> https://github.com/ctriant/GSoC17-Proposal/blob/master/README.md
> https://github.com/ctriant/GSoC17-Proposal/blob/master/gsoc17_proposal.pdf
>
> Thank you in advance,
>
> Kostis
>
>
> _______________________________________________
> Discuss-gnuradio mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

_______________________________________________
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

_______________________________________________
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio