Re: Split Flushing to filter command

[Pádraig Brady]

On 03/20/2014 02:39 AM, Eldon wrote:
> Hello all,
> I am attempting the following in a bash shell on a 3.8.13 linux kernel:
> sudo tcpdump -nn 
>     |grep --line-buffered NTPv2 
>     |split -u --lines=10 --filter=date
> 
> Clearly date would be replaced with some more useful script, but for the
> mean time I am trying to use it to debug what I see as unexpected
> buffering. Since the traffic I am looking at is fairly consistent (when
> I pipe to cat instead, I see a steady stream), I would expect to see
> regular ticks as date executions each time 10 lines are sent to
> split. Instead I see flashes of many executions that seem to me to be a
> buffer flush. I looked in the code in the git repo, and it seems that
> the buffer is in fact filled prior to execution. Is it worth making a
> patch or expanding the meaning of -u to pass this through in an
> unbuffered fashion? Would there be a reason to reject such a patch if
> well-formed?
> 
> Thoughts?

split(1) doesn't use stdio so we're not hitting that buffering.
What's happening is that we're explicitly buffering input
(into a 64K buffer on my x86_64 system) when using full_read() since
coreutils 4.5.8 with this commit:
http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commit;h=23f6d41

Now we really shouldn't be delaying processing like that,
so I propose we switch back to safe_read().

With the attached, the following command line gives immediate output:

while true; do seq 5; sleep 1; done | src/split --lines=5 --filter=date

I'll do a full patch later after checking if safe_read() is
appropriate elsewhere, and adding NEWS and maybe a test.

thanks,
Pádraig.

split-line-buffer.patch
Description: Text Data