Re: date enhancement - filter to preface each line of input with timestamp

[Pádraig Brady]

On 02/11/15 16:47, Evan Rempel wrote:
> On 11/02/2015 02:25 AM, Pádraig Brady wrote:
>> On 01/11/15 22:57, Evan Rempel wrote:
>>> There is a lot of discussion on the net about prefacing a timestamp to
>>> every line of a text stream. There are solutions with awk, perl, python
>>> and bash. coreutils even provides one using perl. This seems to be a
>>> very heavy tool to do something that date already does all of the hard
>>> work for.
>>>
>>> Does it make sense to add an option -p --prefix to make date prefix
>>> every line of stdin with a date/time stamp?
>> It's useful functionality, though already available using other tools.
>> http://stackoverflow.com/q/21564/4421
>> The general rule we use is to not couple logic
>> unless it provides a functional benefit to do so.
>> Personally I like the awk solution presented above.
>>
>> cheers,
>> Pádraig.
> 
> While I agree in principle with the general rule, I do not agree in this 
> case. All of the solutions I can find on the web, including the one your 
> cite, involve using a different programming language to write a program that 
> solves the problem. In some languages, such as awk, the program is very 
> simple, but it still requires invoking another interpreter to execue a 
> program. Using that logic, there isn't any need for most of the coreutils 
> programs. Just use awk to replace cut or perl to replace tr. 
> The coreutils makes this easier to use and easier to read/understand.
> 
> Also in this case, the coreutils package includes an application "ts" that is 
> a perl script that addresses this exact problem. This implies that at some 
> time in the past, coreutils accepted a solution to this problem.

Note `ts` is in the _moreutils_ package.

> I am proposing that this functionality be moved to the date program to make 
> it less dependent on other system tools. The usual use case for this 
> functionality is for log file generation/piping and ths invocation may be 
> prior to these other tools from being available. It may also be on systems 
> that don't have the required tools or at least where these other tools are 
> not consistently available. The coreutils are as the name implies, core, and 
> as such are more consistently available compared to 
> the other more general scripting tools (awk, perl, python).
> 
> This feature would arguably make scripts more readable. I would prefer the 
> later of these two
> 
> <command> | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'
> 
> <command> | date --prefix "+%Y-%m-%d %H:%M:%S"
> 
> To understand what the awk solution does, you have to understand awk.

Yes it's always a tradeoff. Thanks for detailing the options.
Note the fflush() awk bits are a separate part of the problem really,
and coreutils already gives control over that using stdbuf.
So the comparison really is:

<command> | stdbuf -oL awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0 }'

<command> | stdbuf -oL date --prefix '+%Y-%m-%d %H:%M:%S'

Quote close I think.

> Just for effecientcies sake, timing these two solutions on a 1 million line 
> input
> 
> awk
> 1.03user 1.06system 0:02.11elapsed 99%CPU
> 1.11user 1.04system 0:02.17elapsed 99%CPU
> 0.97user 1.17system 0:02.17elapsed 99%CPU
> 
> date --prefix
> 0.63user 0.81system 0:01.48elapsed 97%CPU
> 0.66user 0.80system 0:01.50elapsed 97%CPU
> 0.62user 0.84system 0:01.50elapsed 97%CPU
> 
> Almost 63% more CPU when using awk!

What buffering did you use with the date --prefix test?
How does it compare using: stdbuf -oL date --prefix
I suspect awk might be better than date(1) then
since it's tuned for I/O processing.

> I just wanted to get a feeling for how this feature would be received by the 
> coreutils maintainers before I submitted the actual feature request and patch.

I'm still 60:40 against, given the above options.

If the perl dependency was an issue I'd be more inclined to reimplement ts(1) 
in C
calling into the date libs, and presenting some of the date(1) options,
because that provides additional functionality like relative timestamps.

thanks,
Pádraig.