RE: [ft-devel] Migrating layout table validation code to a new CVSmodule

[Turner, David]

Hello mpsuzuki,

as Werner said, there is no need to apologize. The OpenType and
GX specifications are extremely complex (and underspecified),
and it's very difficult to get this code right.

I'll restate my point that I believe that the validation and
processing code should be part of the same library.

- In theory, separating them is possible when the specification is
  very strict and all font tables obey them to the letter.

- However, in the real world, any non-conformant table that is
  supported by a major system creates an exception to the spec
  that must be coded into/worked around in both libraries.

And you also must ensure that both versions of the libraries
you use on your system are in sync, or subtle problems could
happen; and, of course, this includes security bugs.

A independent table validator is certainly useful for someone
trying to build font files, but I don't think it will be used
by a processor.

That's not to say that the code currently in "ftvalid" couldn't
be integrated in a layout library.

Take care,

- David



> -----Message d'origine-----
> De : address@hidden
> [mailto:address@hidden
> la part de
> address@hidden
> Envoyé : samedi 19 novembre 2005 13:51
> À : Turner, David
> Cc : address@hidden
> Objet : Re: [ft-devel] Migrating layout table validation code to a new
> CVSmodule
>
>
> Hi,
>
> I apologize that I wrote non-negligibly large code in gxvalid
> and it triggered this issue, maybe.
>
> Before all, I have to comment about the important pointing
> out: gxvalid has complexed validator to check indepth data of
> TrueTypeGX tables, but it has bugs, and it does not validate
> more fundamental points, like data alignment, at all.
>
> Although I was aware of alignment issue (see my post on
> 07/Apr/2005), the pointing out is true. AT PRESENT,
> the validation result of gxvalid does not mean TrueTypeGX
> font is correctly structured. When working with gxvalid,
> I found that Apple bundles TrueTypeGX fonts including
> incorrect GX tables and Apple's implementation stands
> against them and works. In addition to, in the working
> of gs-cjk project (http://www.gyve.org/gs-cjk/), we
> already knew that there are many TrueType fonts with bad
> alignment issue.
>
> Unfortunately, most popular platforms of OpenType and
> TrueTypeGX (MS Windows and MacOS) are running on the
> architechtures that unaligned memory access does not
> cause severe problem. From the pragmatical viewpoint,
> I thought the early revision of gxvalid should check
> loosely (at least, the fonts bundled to MacOS should
> be passed as OK), and I should improve it to validate
> more strictly. Why from loose to strict? If I start
> from the severe validation, I thought, it will mark
> most of exisiting TrueTypeGX fonts as "broken", and
> it can discourage from utilization of TrueTypeGX. So,
> I thought it's better to start from loose to strict.
>
> Turner, how do you think of? We should start from
> strict and adding capability for exceptional cases?
> Or, unaligned memory access is different issue and
> should be validated strictly from initial revision?
>
> After the initial commit of gxvalid to CVS, to improve
> gxvalid, I prioritize working for MacOS port of FreeType,
> especially for m68k classic MacOS (which is the hometown
> of TrueTypeGX). That's why I could not reflect the
> important comments by George Williams and could not fix
> the parts of gxvalid that you marked XXX, sorry for too
> late to explain.
>
> On Fri, 18 Nov 2005 14:48:30 +0100
> "Turner, David" <address@hidden> wrote:
> >> BTW, I'm not really happy that you want to move it into another CVS
> >> module.  Those files are really part of FreeType and quite tightly
> >> integrated.  Instead I suggest that we extend FreeType's
> source tree
> >> structure, adding a new top-level directory `addons' which
> is filled
> >> with extra stuff.
> >>
> >First, I don't consider this code to be part of the font engine. It
> >does nothing for anyone trying to load and render fonts at 
> the moment.
> >
> >It can only be useful in conjunction with a higher-level text layout
> >library.
>
> >As said earlier, this code can only be used in conjuction with a
> >higher-level text layout library that we're certain not to provide
> >with FT2.
> >
> >The problem is that if both the layout library and validation code
> >are written separately, there is the possibility of a mismatch, where
> >what "ftvalid" think is valid might not be appropriate for the layout
> >one.
>
> I think the validation itself should be independent
> from layout engine.
>
> When I was working for gxvalid, the most standard
> usage of gxvalid would be a part of font file utility
> (font editor, font manager, etc), than as a part of
> text layout engine.
>
> In fact, Apple provides several font validators
> (latest Mac OS X 10.4.x includes a validator by default,
> and uses it to blockade the installation of broken font),
> but their validation results are incompatible among
> the tools, and less informative about the detail of
> breaking, so the font developers might be frustrated.
> I think open source and verbose font validator is required.
>
> Either Microsoft has another validator - it might be
> different from Apple's validators and possibly its
> validation differs from Microsoft text layout engine's
> capabilities. In fact, I heard that Japanese free fonts
> are known to be installable into MS Windows font folder
> and cause no severe problems, but Microsoft validator
> finds many incorrect data in it.
>
> Another reason is that: if the text layout engine includes
> its own validator, it will be runtime checking to avoid
> from wrong behaviour and crashing, and won't validate
> unaccessed parts. It's not easy to use text layout functions
> to crawl all features declared in the OpenType/TrueTypeGX.
>
> >If you find a security bug in the layout library, you need to ensure
> >that it is also fixed in the validation code. And vice-versa... This
> >is a hassle.
>
> Umm, do you mean something like: if gxvalid returns false
> "conforming" result to ICU, it can cause security bug in
> ICU, so ICU won't believe the result of gxvalid and ICU
> will have their own validator...?
>
> >> > I guess that we will not remove the public APIs defined
> so far from
> >> > FreeType 2, but their implementation will simply return
> >> > FT_Err_Unimplemented instead.
> >>
> >> Ideally, FreeType should have support for external
> plugins.  This is,
> >> if FreeType finds libftvalid, it uses it, otherwise you get
> >> FT_Err_Unimplemented.  It seems that we have to start with
> libtool's
> >> `dlopen' function...
> >>
> >> Sigh.  All these things mean a lot of additional work.  Isn't it
> >> easier to simply disable those two modules?  I'm already
> thinking how
> >> to have a central configuration file which controls the
> used modules
> >> -- without removing or renaming the rules.mk files...
> >>
> >We probably need something like that, but I want, at a minimum, that
> >these modules be out of a default build.
>
> About plugin, it is good idea (although I'm afraid
> that FreeType is very very widely used on platforms
> without dlopen etc), but it will be long way work.
> It should be FreeType3, even if the binary compatibility
> is kept.
>
> I have no objection against the pointing out: gxvalid
> is too big as a module built/included by default.
> I think, otlayout is still very compact in contrast
> with gxvalid, and no problem to include it in freetype.
> If I rewrite gxvalid to smaller and limited validation
> (e.g. memory alignment, number of members etc) by
> default, it's acceptable to include?
>
> Regards, sorry for long mail with poor English.
> mpsuzuki
>
>
> _______________________________________________
> Freetype-devel mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/freetype-devel
>
***********************************************************************************
Information contained in this email message is confidential and may be 
privileged, and is intended only for use of the individual or entity named 
above. If the reader of this message is not the intended recipient, or the 
employee or agent responsible to deliver it to the intended recipient, you are 
hereby notified that any dissemination, distribution or copying of this 
communication is strictly prohibited. If you have received this communication 
in error, please immediately notify the address@hidden and destroy the original 
message.
***********************************************************************************