Re: [Nano-devel] unicode test with nano

[David Lawrence Ramsey]

Jordi Mallach wrote:

<snip>

>David, this is one of the most asked questions regarding nano's UTF-8
>support. Do you think we could do cheats like vim in 1.6, or would that
>be a major rewrite of the core?

Possibly.  Some parts may need a major rewrite, depending on what gets
done.  The difficult part is the actual encoding conversion.  Either
nano follows in the footsteps of other editors I've seen and includes a
lot of extra internal code to convert between a hardcoded list of
encodings, thus duplicating a lot of work already done elsewhere; or it
uses iconv() to do the conversion, where (I've heard) different
implementations of it are of varying quality, can't always handle all
common encodings, and can result in data loss if an exact conversion
between encodings isn't possible.

If we go the iconv() route, then we'll probably have to do what other
iconv()-using programs do and store everything internally as wide
characters with Unicode values, which may be problematic on older
systems that don't support wide characters (which is exactly why nano
1.3.x handles multibyte characters instead of wide characters).  Not to
mention, such systems may not have iconv() at all.  I believe that the
GNU project has an implementation of it meant to be used in other
projects (although I forget its name), but if it's anything like the
snprintf() functions provided by gnulib, it would make things horribly
messy.

Incidentally, this last item is why I haven't switched to the gnulib
version of vsnprintf() for nano, although I did look into it once some
months ago.  Current CVS doesn't need snprintf() anymore because I found
out that it was only used when printing one literal null-terminated
string, and that was easily enough replaced with a charalloc(strlen() +
1) and an sprintf().  Adding the gnulib code would require, among other
things, adding a ton of new m4 files to handle all the types that the
printf() functions do and most of which nano doesn't use, manually
removing bad dependencies in the code modules (Why does the vsnprintf
module need the minmax module, since it doesn't use the min() or max()
macros anywhere?  Why is the asnprintf module included, since neither it
nor any of its dependencies are used by the vsnprintf module or the
modules that it includes?  Why is alloca(), or at the very least a stub
alloca.h, used by default in builds that aren't part of libintl, even
though the former is a nonstandard function?), making the
autoconf-related files even more screwed up than they already are by
separately handling all the aforementioned m4 files and adding library
linking (Why does it need to handle its own m4 files before all other m4
files?  Why does it need to put all its functions in a static library
and then require linking to it later?  Why isn't there an easy way to
build it on systems that have the functions, for testing purposes
only?), and then expecting it to build everywhere under the same
conditions that nano does.  If you're using ncurses, nano will even
build properly, albeit with some unavoidable warnings about
signed/unsigned comparisons (since LINES and COLS are stored in signed
variables) if you use g++ as the compiler and -pedantic as one of your
flags; slang builds will fail because the warning about converting const
char*'s to char*'s in C becomes an error in C++.