Given platforms tcc supports, I think you can assume, wchar_t uses 2 bytes on
Windows and 4 bytes on all other platforms (I'm not totally sure, but think you
can force wchar_t to be 2 bytes on macOS).
I've never heard about other implementation for wchar_t (I don't recall how zOS
encodes wchar_t but I doubt someone will port tcc on this system which still
uses EBCDIC natively).
Le : 01 septembre 2017 à 11:02 (GMT +02:00)
De : "张博洋" <address@hidden>
À : "address@hidden" <address@hidden>
Objet : Re: [Tinycc-devel] BUG: wide char in wide string literal handled
incorrectly
Hello,
Thanks for your reply.
My assumptions only applicable to wide string literals. The behavior for
plain strings literals of both original tcc and my patched tcc is "copy
bytes in plain string as is". And for wide strings, original tcc "read
each char and cast them to wchar_t", my patched tcc "decode them as
UTF-8 sequences".
After some consideration, I found the assumption I made was "wide string
literals are written in UTF-8, and wchar_t is always UTF-32". That leads
to two problems. First, wide string encoding in source file is
definitely same as the encoding of source file, which might not be
UTF-8. This will cause problems as you mentioned. Second, wchar_t is not
always UTF-32. It's UTF-16 on Microsoft Windows. Some chars, like emojis
, will get corrupted because of value truncation.
Although there are problems, if the second problem got fixed (which is
easy), my patched tcc will always perform better than original tcc. If
something breaks, it will also breaks on original tcc. I provided a
table in attachments describing every situation and corresponding behaviors.
The ideal solution is to provide charset options as you mentioned. After
doing some search on internet, I found that there are 3 command line
options that controls char encoding:
-fexec-charset=charset
-fwide-exec-charset=charset
-finput-charset=charset
In order to make these feature works correctly, tcc must do two conversions:
(1) convert all plain string literal from input-charset to exec-charset
(2) convert all wide string literal from input-charset to wide-exec-charset
However, providing these feature requires external libraries like iconv,
doing this might make Tiny C Compiler not tiny.
My problems are:
(1) Is wchar_t either UTF-32 or UTF-16 on all platforms?
(2) Should we provide full support for charset using external librarys?
Thanks
Zhang Boyang
在 2017年09月01日 11:54, Christian Jullien 写道:
> Hello,
>
> I'm not sure you can assume that a character having code >= 0x80 is part of UTF-8. Beyond
what is called "basic character set" which is globally the ASCII 7bits, there is the
"extended character set" which is implementation defined.
>
> For example, the euro sign EUR may be part of 8859-15 and perfectly well
encoded on 8bits with 0xA4 see https://en.wikipedia.org/wiki/ISO/IEC_8859-15
>
> Microsoft VC++ has the following flags
>
> /utf-8 set source and execution character set to UTF-8
> /validate-charset[-] validate UTF-8 files for only legal characters
>
> That controls how source code is encoded.
>
> gcc (more specifically cpp the C preprocessor) processes source file using
UTF-8 but, as VC++ has a flag to control input-char
>
> -finput-charset=charset
> Set the input character set, used for translation from the
> character set of the input file to the source character set
used by
> GCC. If the locale does not specify, or GCC cannot get this
> information from the locale, the default is UTF-8. This can be
> overridden by either the locale or this command-line option.
> Currently the command-line option takes precedence if there's a
> conflict. charset can be any encoding supported by the
system's
> "iconv" library routine.
>
> Now, tcc should be compatible with both. I mean:
>
> - Native Windows tcc port should NOT assume characters are UTF-8 encoded
and -utf-8 flag should change this behavior (+ -finput-charset=xxx for gcc
compatibility)
> - Other ports (I mean Linux & alt.) should assume characters are UTF-8
encoded and -finput-charset=xxx flag should change this behavior (+ -utf-8 for VC++
compatibility)
>
> To summarize, which should add both utf-8 and -finput-charset=xxx support
and set the default behavior based on native port.
>
> Wdyt?
>
> Christian
>
>
> -----Original Message-----
> From: Tinycc-devel [mailto:address@hidden On Behalf Of ???
> Sent: mercredi 30 août 2017 09:31
> To: address@hidden
> Subject: [Tinycc-devel] BUG: wide char in wide string literal handled
incorrectly
>
> Hello,
>
> I found that when TCC processing wide string literal, it behaves like
directly casting each char in original file to wchar_t and store them in wide
string. This will work for ASCII chars. However, it might not work for real wide
chars. For example:
> The Euro-sign (EUR, U+20AC) stored in UTF-8 is "E2 82 AC". In GCC, this char stored in
wide string will be "000020AC". However, in TCC, this char is stored as 3 wide chars "000000E2
00000082 000000AC".
> I provided a patch, a test program and two screenshots that describe this
problem, they are in attachments. I solve this problem by making assumptions that input
charset is UTF-8. Although it's not a perfect solution, it's still better than "directly
casting char to wchar_t". I'm wondering if that is appropriate, so please review the
code carefully.
>
> Thanks
> Zhang Boyang
>
>
> _______________________________________________
> Tinycc-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/tinycc-devel
>
_______________________________________________
Tinycc-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
_______________________________________________
Tinycc-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
utf8-new.patch