Re: [ft-devel] Migrating layout table validation code to a new CVS module

[mpsuzuki]

Hi,

I apologize that I wrote non-negligibly large code in gxvalid 
and it triggered this issue, maybe.

Before all, I have to comment about the important pointing
out: gxvalid has complexed validator to check indepth data of
TrueTypeGX tables, but it has bugs, and it does not validate
more fundamental points, like data alignment, at all.

Although I was aware of alignment issue (see my post on
07/Apr/2005), the pointing out is true. AT PRESENT,
the validation result of gxvalid does not mean TrueTypeGX
font is correctly structured. When working with gxvalid,
I found that Apple bundles TrueTypeGX fonts including
incorrect GX tables and Apple's implementation stands
against them and works. In addition to, in the working
of gs-cjk project (http://www.gyve.org/gs-cjk/), we
already knew that there are many TrueType fonts with bad
alignment issue.

Unfortunately, most popular platforms of OpenType and
TrueTypeGX (MS Windows and MacOS) are running on the
architechtures that unaligned memory access does not
cause severe problem. From the pragmatical viewpoint,
I thought the early revision of gxvalid should check
loosely (at least, the fonts bundled to MacOS should
be passed as OK), and I should improve it to validate
more strictly. Why from loose to strict? If I start
from the severe validation, I thought, it will mark
most of exisiting TrueTypeGX fonts as "broken", and
it can discourage from utilization of TrueTypeGX. So,
I thought it's better to start from loose to strict.

Turner, how do you think of? We should start from
strict and adding capability for exceptional cases?
Or, unaligned memory access is different issue and
should be validated strictly from initial revision?

After the initial commit of gxvalid to CVS, to improve
gxvalid, I prioritize working for MacOS port of FreeType,
especially for m68k classic MacOS (which is the hometown
of TrueTypeGX). That's why I could not reflect the
important comments by George Williams and could not fix
the parts of gxvalid that you marked XXX, sorry for too
late to explain.

On Fri, 18 Nov 2005 14:48:30 +0100
"Turner, David" <address@hidden> wrote:
>> BTW, I'm not really happy that you want to move it into another CVS
>> module.  Those files are really part of FreeType and quite tightly
>> integrated.  Instead I suggest that we extend FreeType's source tree
>> structure, adding a new top-level directory `addons' which is filled
>> with extra stuff.
>> 
>First, I don't consider this code to be part of the font engine. It
>does nothing for anyone trying to load and render fonts at the moment.
>
>It can only be useful in conjunction with a higher-level text layout
>library.

>As said earlier, this code can only be used in conjuction with a
>higher-level text layout library that we're certain not to provide
>with FT2.
>
>The problem is that if both the layout library and validation code
>are written separately, there is the possibility of a mismatch, where
>what "ftvalid" think is valid might not be appropriate for the layout
>one.

I think the validation itself should be independent
from layout engine.

When I was working for gxvalid, the most standard
usage of gxvalid would be a part of font file utility
(font editor, font manager, etc), than as a part of
text layout engine.

In fact, Apple provides several font validators
(latest Mac OS X 10.4.x includes a validator by default,
and uses it to blockade the installation of broken font),
but their validation results are incompatible among
the tools, and less informative about the detail of
breaking, so the font developers might be frustrated.
I think open source and verbose font validator is required.

Either Microsoft has another validator - it might be
different from Apple's validators and possibly its
validation differs from Microsoft text layout engine's
capabilities. In fact, I heard that Japanese free fonts
are known to be installable into MS Windows font folder
and cause no severe problems, but Microsoft validator
finds many incorrect data in it.

Another reason is that: if the text layout engine includes
its own validator, it will be runtime checking to avoid
from wrong behaviour and crashing, and won't validate
unaccessed parts. It's not easy to use text layout functions
to crawl all features declared in the OpenType/TrueTypeGX.

>If you find a security bug in the layout library, you need to ensure
>that it is also fixed in the validation code. And vice-versa... This
>is a hassle.

Umm, do you mean something like: if gxvalid returns false
"conforming" result to ICU, it can cause security bug in
ICU, so ICU won't believe the result of gxvalid and ICU
will have their own validator...?

>> > I guess that we will not remove the public APIs defined so far from
>> > FreeType 2, but their implementation will simply return
>> > FT_Err_Unimplemented instead.
>> 
>> Ideally, FreeType should have support for external plugins.  This is,
>> if FreeType finds libftvalid, it uses it, otherwise you get
>> FT_Err_Unimplemented.  It seems that we have to start with libtool's
>> `dlopen' function...
>>
>> Sigh.  All these things mean a lot of additional work.  Isn't it
>> easier to simply disable those two modules?  I'm already thinking how
>> to have a central configuration file which controls the used modules
>> -- without removing or renaming the rules.mk files...
>> 
>We probably need something like that, but I want, at a minimum, that
>these modules be out of a default build.

About plugin, it is good idea (although I'm afraid
that FreeType is very very widely used on platforms
without dlopen etc), but it will be long way work.
It should be FreeType3, even if the binary compatibility
is kept.

I have no objection against the pointing out: gxvalid
is too big as a module built/included by default.
I think, otlayout is still very compact in contrast
with gxvalid, and no problem to include it in freetype.
If I rewrite gxvalid to smaller and limited validation
(e.g. memory alignment, number of members etc) by
default, it's acceptable to include?

Regards, sorry for long mail with poor English.
mpsuzuki