RE: Re: Getting GPS data into stream

[Jim Melton]

You are confusing data representation with data presentation. The flaws in 
Excel are NOT issues with the data format. So long as the data format clearly 
and consistently represents that content, then the representation is "good".

If you want to overcome limitations in Excel's presentation (import, 
interpretation), then that's an Excel issue. You can overcome it by manually 
doing the import and explicitly asserting the data type of each column, or you 
can create something more custom.

Realize that more and more your data is likely to be consumed by some other 
data science tool (R, Python/numpy/pandas, etc.) and you quickly see how 
pushing Excel issues into the data representation layer is a losing proposition.

---
Jim Melton


-----Original Message-----
Sent: Thursday, May 4, 2023 01:40
To: discuss-gnuradio@gnu.org
Subject: [EXTERNAL] Re: Getting GPS data into stream

Hey Marcus,

as you say, for a lot of science you don't get high rates – so I'm really less 
worried 
about that. More worried about Excel interpreting some singular data point as 
date; or, as 
soon as we involve textual data, all the funs with encodings, 
quoting/delimiting/escaping… 
(not to mention that an Excel set to German might interpret different things as 
numbers 
than a Northern American one).

I wish there was just one good CSV standard that tools adhered to. Alas, that's 
not the 
case, and especially Excel has a habit of autoconverting input and losing data 
at that point.
So, looking for an alternative that has these well-defined constraints and 
isn't as 
focused on hierarchical data (JSON, YAML, XML), far too verbose but excellent 
to query 
with command line tools (XML), completely impossible to correctly parse as 
human or parser 
in its full beauty (YAML)… Just some tabular data notation that's textual, 
appendable, and 
not a party of guesswork for the reading tool.
We could just canonalize calling all our files

marcusdata.utf8.textalwaysquoted.iso8601.headerspecifies_fieldname_parentheses_type.csv

but even that wouldn't solve the issue of excel seeing an unquoted 12.2021 and 
deciding 
the field being about christmases past.

So, maybe we just do some rootless JSON format that starts with a SigMF object 
describing 
the file and its columns, and then basically is just a sequence of JSON arrays

[ 1.212e-1, 0, "Müller", 24712388823 ]
[ 1.444e-2, 1, "📡🔭  \"👽\"!", 11111111111 ]
[ 2.0115-1, 0, "Cygnus-B", 0 ]

(I'm not even sure that's not valid JSON; gut feeling tells me we should be 
putting [] 
around the whole document, but we don't want that for streaming purposes. 
ECMA-404 doesn't 
seem to *forbid* it.)

That way, we get the metadata in a format that's easy to skip by simpler tools, 
but 
trivial to parse with the right tools (I've grown to like `jq`), and the data 
into a 
well-defined format. Sure, you can't dump that into Excel, still, but you know 
what, if it 
comes down to it, we can have a python script that takes these files and 
actually converts 
them to valid XLSX without the misconversion footguns, and that same tool could 
also be 
run in a browser for those having a hard time executing python on their 
machines.

Cheers,
Marcus
On 03.05.23 23:05, Marcus D. Leech wrote:
> On 03/05/2023 16:51, Marcus Müller wrote:
>>
>> Do agree, but really don't like CSV, too underspecified a format, too many 
>> ways that 
>> comes back to bite you (aside from a thousand SDR users writing emails that 
>> their PC 
>> can't keep up with writing a few MS/s of CSV…)
> I like CSV because you can hand your data files to someone who doesn't have a 
> complete 
> suite of astrophysics tools, and they
>    can slurp it into Excel and play with it.
> 
>>
>> How important is plain-textness in your applications?
> I (and many others in my community) tend to throw ad-hoc tools at data from 
> ad-hoc 
> experiments.  In the past, I used a lot
>    of AWK to post-process data, and these days, I use a lot of Python.    
> Text-based 
> formats lend themselves well to this kind
>    of processing.  Rates are quite low, typically.  Like logging an 
> integrated power 
> spectrum a few times a minute, for example.
> 
> There are other observing modes where text-based formats aren't quite so 
> obvious--like 
> pulsar observations, where filterbank
>    outputs might be recorded at 10s of kHz, and then post-processed with any 
> of a number 
> of pulsar tools.
> 
> In all of this, part of the "science" is extracted in "real-time" and part in 
> post-processing.
> 
> 
>>
>> Best,
>> Marcus
>>
> 
> 

CONFIDENTIALITY NOTICE - SNC EMAIL: This email and any attachments are 
confidential, may contain proprietary, protected, or export controlled 
information, and are intended for the use of the intended recipients only. Any 
review, reliance, distribution, disclosure, or forwarding of this email and/or 
attachments outside of Sierra Nevada Corporation (SNC) without express written 
approval of the sender, except to the extent required to further properly 
approved SNC business purposes, is strictly prohibited. If you are not the 
intended recipient of this email, please notify the sender immediately, and 
delete all copies without reading, printing, or saving in any manner. --- Thank 
You.

smime.p7s
Description: S/MIME cryptographic signature