Re: [bug-gnu-libiconv] iconv tool buffers entire input file

[Bruno Haible]

Hi,

Yongzheng Wu wrote:
> The iconv tool buffers the entire input file in memory. This causes a
> problem with big files. This problem has been raised by others before.
> http://serverfault.com/questions/50919/how-to-run-command-on-parts-of-input-file
> and http://keithdevens.com/weblog/archive/2011/Sep/22/iconv. I found a
> comment in iconv_prog.c, "Since we have to deal with arbitrary
> encodings we must read the whole text in a buffer and process it in
> one step."

Your report is about the 'iconv' program from GNU libc. Here we are on the
mailing list for GNU libiconv, which is a different package. The 'iconv'
program from GNU libiconv does not have the behaviour you report.

> My question is, can this be fixed?

Yes, this can be fixed, with fixed-size buffers and careful handling of
the unprocessed bytes at the end of a block.

I encourage you to report this to the GNU libc bug tracker, at
http://sourceware.org/bugzilla/

Bruno