Re: [PATCH] Fixed incomplete and incorrect treatment of comments and trailing whitespace

[Erik Auerswald]

Hi Dima,

On Thu, May 19, 2022 at 11:47:44PM -0700, Dima Kogan wrote:
> Erik Auerswald <auerswal@unix-ag.uni-kl.de> writes:
> 
> > From a quick glance at the code diff in the link, this seems to allow
> > comments inside a field, e.g., with datamash -H -C -t',' and the
> > following input:
> >
> >     # the next line is the header line
> >     one,two,three
> >     # the following line is the data line
> >     1,2#this is the 2nd field,3
> >
> > in the data line the string "#this is the 2nd field" would be skipped,
> > and the data line would have three fields with values 1, 2, and 3.
> >
> > Is that correct (I did not test it)? Is that the intended
> > functionality?
> 
> I haven't tested this yet, but that wasn't my intent. Comments should do
> what they do in perl and python and awk and everywhere else, so the last
> line should be interpreted as "1,2". I may have made a mistake in the
> implementation.

I have just tested it, this is not the way you're code behaves.
Everything beginning with the comment sign is ignored, as intended.

I misread the diff.

Just to be clear: I also would expect this behavior, i.e., the comment
                  sign and everything until the end of the line is
                  ignored as a comment.

I'd still prefer this change to comment handling to be off by default
and only activated with a new option.  E.g., as an option that modifies
-C behavor by adding support for in-line comments, or as an option that
ignores only in-line comments (I'd prefer the former).  Modifying the -C
behavior could automatically enable -C if it was not given explicitly,
to avoid having to add two options for the intended behavior.

Thanks,
Erik