Re: [Devel] Half-width ASCII in monospace CJK fonts

[David Turner]

Hello Anthony,

Anthony Fok wrote:

> Hello David and Werner,
>
> Quite a few popular CJK fonts, including some latest offerings from
> Arphic (e.g. HKSCS-2001 fonts published in 2002), have
> face->postscript.isFixedPitch set to "true" to mean fixed full-width
> Hanzi characters, and fixed half-width ASCII/Latin characters (vs. some
> CJK fonts with proportional width ASCII/Latin characters where
> face->postscript.isFixedPitch is set to "false".)
>
> This often lead to some problems when using such fonts in CJK locales.
> For example, a menu bar normally displayed as:
>
>    [ File  Edit  View                        Help ]
>
> becomes
>
>    [  F i l e    E d i t     V i e w         H e l p  ]
>
> even when FreeType 2 is used for rendering.  (i.e. not the X11 XLFD *-c-*
> charcell font problem.)
>  
>
> This issue has been brought up before by Graham Asher, but
> unfortunately, he didn't get very far.  See the thread starting here:
>  
>
>    http://www.freetype.org/pipermail/devel/2001-March/001768.html
>  
It seems I've missed this one, sorry Graham, I hope the following answer 
will suffice for you too.

> I do not advocate ignoring the postscript.isFixedPitch flag.  Rather, I
> propose reversing the default of "FT_LOAD_IGNORE_GLOBAL_ADVANCE_WIDTH".
> Let me explain.  With respect to the following comment in the current
> freetype.h:
>
>  *   FT_LOAD_IGNORE_GLOBAL_ADVANCE_WIDTH ::
>  *     Indicates that the glyph loader should ignore the global advance
>  *     width defined in the font.  As far as we know, this is only used by
>  *     the X-TrueType font server, in order to deal correctly with the
>  *     incorrect metrics contained in DynaLab's TrueType CJK fonts.
>
> I do not entirely agree with the above explanation.  I do not have
> DynaLab's TrueType CJK fonts at hand, but I believe that the use of
> postscript.isFixedPitch in a TrueType font's "post" table to indicate
> "Fixed half-width ASCII/Latin chars, Fixed full-width CJK chars" is
> quite reasonable and not "incorrect".  Furthermore, I believe that it
> is reasonable to trust the font's own advance width for each glyph to
> be correct.
>  
Well, I think you're getting things backwards. When FreeType first came 
out, it only used
to set the FT_FACE_FLAG_FIXED_WIDTH bit flag according to the content of 
the font
headers, and loaded each glyph "normally", i.e. by using the advance 
width specified in each
glyph record.

Fact is that this gave surprising results, mostly because many of the 
popular mono-spaced fonts
available have glyphs with advances that do *not* correspond to the 
global advance width.

This resulted in extremely  unpleasant artefacts (like the inability to 
properly display anti-aliased
text in consoles, because of mis-alignments caused by the "incorrect" 
advances). Hence, FT2's
behaviour has been modified to always use the global advance width when 
the FIXED_WIDTH
flag is set. Of course, we also provided 
FT_LOAD_IGNORE_GLOBAL_ADVANCE_WIDTHS
for the cases where the original metrics were needed (and to date, that 
was the X-TrueType
server exclusively).

In short, we simply _cannot_  "trust" the font's content. Amusing, isn't 
it :-)

I have written a small program named "ftcheckfixedwidth" that is no in 
the "ft2demos" of the
CVS repository, in case you'd like to take a look at it (you need to 
manually compile it,
unless you're using Jam, but this shouldn't be too hard since it's just 
a modification of the
"ftlint" test program)

You should try on it on all the fonts of your system. On mine, I've got 
the interesting
following results:

 * Courier New and Andale Mono both have 1 "incorrect" glyph (the space 
it seems)
 * All glyphs in Lucida Console and Lucidux Mono have variable advances

You can now imagine that we cannot simply revert FreeType's behaviour to 
its original
one without breaking text layout for *many* users :o)

> So, in my opinion, FreeType should ignore the global advance width by
> default, and should only force global advance width only if explicitly
> demanded.  It would be nice to see something like a new
>
>         FT_LOAD_FORCE_GLOBAL_ADVANCE_WIDTH
>                ^^^^^
> to supercede FT_LOAD_IGNORE_GLOBAL_ADVANCE_WIDTH.
>
> Anyhow, it would also be nice, when the "force global advance width"
> is on, that FreeType could respect the distinction between "full-width"
> and "half-width" as described in
>
>         Unicode Standard Annex #11 East Asian Width
>         http://www.unicode.org/unicode/reports/tr11/
>
> and  http://www.unicode.org/Public/3.2-Update/EastAsianWidth-3.2.0.txt
> But yes, that does add a bit of complexity to FreeType.  Hmm...
>  
I tend to believe that this is the problem of the text layout engine, 
not of FreeType itself.
One can analyze the charmaps provided to detect most half-width and 
full-width forms.
There is still the case of ambiguous characters that cannot be 
explicited without additional
context that only the layout layer knows about anyway.

Of course, let me know if you have a good argument that would prove that 
adding such
support within the engine would be better.

> Anyhow, please do seriously consider reversing the default setting, and
> implementing something similar to FT_LOAD_FORCE_GLOBAL_ADVANCE_WIDTH.
> I have been using a kludgy patch in the FreeType package on Thiz Linux
> and on Debian to get around this problem.  (See attachments... please
> do not apply them; they are ugly hacks)  But yes, it would be nice if
> this problem could be solved in FreeType for good.  :-)
>  
I will certainly not _reverse_ the behaviour. However, it may be 
possible to make FT2 a bit more
intelligent about the way it computes glyph advance widths. Certainly, 
there must be a better way
than simply forcing the global advance width when a single header bit 
flag is set.

However, I have no idea for the moment of any "smarter" algorithm (maybe 
one that looks at
other tables in the file, but which ones precisely). Keep in mind that 
any alternative solution should
work on the *vast* majority of available fonts today.

Until we find this "better" way, I won't change the engine's behaviour...

Sorry,

- David Turner
- The FreeType Project  (www.freetype.org)