bug-gnu-libiconv
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[bug-gnu-libiconv] iconv tool buffers entire input file


From: Yongzheng Wu
Subject: [bug-gnu-libiconv] iconv tool buffers entire input file
Date: Fri, 7 Dec 2012 00:21:35 +0800

Hi,

The iconv tool buffers the entire input file in memory. This causes a
problem with big files. This problem has been raised by others before.
http://serverfault.com/questions/50919/how-to-run-command-on-parts-of-input-file
and http://keithdevens.com/weblog/archive/2011/Sep/22/iconv. I found a
comment in iconv_prog.c, "Since we have to deal with arbitrary
encodings we must read the whole text in a buffer and process it in
one step."

My question is, can this be fixed? A trivial approach is to process
the input in fixed-size blocks. The remaining unprocessed bytes of
each block is feed to the next block. I think this approach works with
all prefix code encodings including UTF-8, UTF-16 and GBK.

Wu Yongzheng



reply via email to

[Prev in Thread] Current Thread [Next in Thread]