[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Bash scripting and large files: input with the read builtin from a r
From: |
Chet Ramey |
Subject: |
Re: Bash scripting and large files: input with the read builtin from a redirection gives unexpected result with files larger than 2GB. |
Date: |
Sun, 04 Mar 2012 21:04:01 -0500 |
User-agent: |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 |
On 3/2/12 6:47 AM, Jean-François Gagné wrote:
> Description:
> When reading data with the 'read' buildin from a redirection, read has
> unexpected behavior after reading 2G of data.
>
> Repeat-By:
>
>
> yes "0123456789abcdefghijklmnopqrs" | head -n 100000000 > file
> while read line; do file=${line:0:10}; echo $file; done < file | uniq -c
>
>
> results in
>
>
> 71582790 0123456789
> 1 mnopqrs
> 3 0123456789
> 1 mnopqrs
> 3 0123456789
> 1 mnopqrs
> 3 0123456789
> 1 mnopqrs
> 3 0123456789
> ...
>
> So the problem happens after reading 71.582.790 x30 = 2.147.483.700 bytes of
> data, just a little over 2^31.
>
> but the following:
>
> cat file | while read line; do file=${line:0:10}; echo $file; done | uniq -c
>
> works fine:
>
> 100000000 0123456789
This works fine with the patch I posted.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU chet@case.edu http://cnswww.cns.cwru.edu/~chet/
- Bash scripting and large files: input with the read builtin from a redirection gives unexpected result with files larger than 2GB., Jean-François Gagné, 2012/03/03
- Re: Bash scripting and large files: input with the read builtin from a redirection gives unexpected result with files larger than 2GB., Chet Ramey, 2012/03/04
- Re: Bash scripting and large files: input with the read builtin from a redirection gives unexpected result with files larger than 2GB., Jonathan Nieder, 2012/03/04
- Re: Bash scripting and large files: input with the read builtin from a redirection gives unexpected result with files larger than 2GB., Bob Proulx, 2012/03/04
- Re: Bash scripting and large files: input with the read builtin from a redirection gives unexpected result with files larger than 2GB., Andreas Schwab, 2012/03/04
- Re: Bash scripting and large files: input with the read builtin from a redirection gives unexpected result with files larger than 2GB., Chet Ramey, 2012/03/04
- Re: Bash scripting and large files: input with the read builtin from a redirection gives unexpected result with files larger than 2GB., Chet Ramey, 2012/03/04
- Re: Bash scripting and large files: input with the read builtin from a redirection gives unexpected result with files larger than 2GB., Andreas Schwab, 2012/03/04
- Re : Bash scripting and large files: input with the read builtin from a redirection gives unexpected result with files larger than 2GB., Jean-François Gagné, 2012/03/06
- Re : Bash scripting and large files: input with the read builtin from a redirection gives unexpected result with files larger than 2GB., Jean-François Gagné, 2012/03/08
Re: Bash scripting and large files: input with the read builtin from a redirection gives unexpected result with files larger than 2GB.,
Chet Ramey <=