Re: [bug-gnu-libiconv] iconv tool buffers entire input file

[Jeffrey Walton]

On Thu, Dec 6, 2012 at 8:13 PM, Bruno Haible <address@hidden> wrote:
> Hi,
>
> Yongzheng Wu wrote:
>> The iconv tool buffers the entire input file in memory. This causes a
>> problem with big files. This problem has been raised by others before.
>> http://serverfault.com/questions/50919/how-to-run-command-on-parts-of-input-file
>> and http://keithdevens.com/weblog/archive/2011/Sep/22/iconv. I found a
>> comment in iconv_prog.c, "Since we have to deal with arbitrary
>> encodings we must read the whole text in a buffer and process it in
>> one step."
>
> Your report is about the 'iconv' program from GNU libc. Here we are on the
> mailing list for GNU libiconv, which is a different package. The 'iconv'
> program from GNU libiconv does not have the behaviour you report.
>
>> My question is, can this be fixed?
>
> Yes, this can be fixed, with fixed-size buffers and careful handling of
> the unprocessed bytes at the end of a block.
>
> I encourage you to report this to the GNU libc bug tracker, at
> http://sourceware.org/bugzilla/
Good catch Bruno.

I spent last night and this morning thinking about gaps in my negative
test cases, and why I did not see a problem.

Jeff