[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[bug-gnu-libiconv] iconv tool buffers entire input file
From: |
Yongzheng Wu |
Subject: |
[bug-gnu-libiconv] iconv tool buffers entire input file |
Date: |
Fri, 7 Dec 2012 00:21:35 +0800 |
Hi,
The iconv tool buffers the entire input file in memory. This causes a
problem with big files. This problem has been raised by others before.
http://serverfault.com/questions/50919/how-to-run-command-on-parts-of-input-file
and http://keithdevens.com/weblog/archive/2011/Sep/22/iconv. I found a
comment in iconv_prog.c, "Since we have to deal with arbitrary
encodings we must read the whole text in a buffer and process it in
one step."
My question is, can this be fixed? A trivial approach is to process
the input in fixed-size blocks. The remaining unprocessed bytes of
each block is feed to the next block. I think this approach works with
all prefix code encodings including UTF-8, UTF-16 and GBK.
Wu Yongzheng
- [bug-gnu-libiconv] iconv tool buffers entire input file,
Yongzheng Wu <=