[Devel] Regarding the layout table formats & FreeType 2

[David Turner]

Hello,

I've completed my analysis of Masatake YAMATO's code for handling
the OpenType Layout & Apple GX/AAT tables and would like to share my
opinion about it with you.

First of all, I'm impressed by the work, developing this code was
a rather important task, and I thank Masatake for contributing it.
There is little doubt that what's in there has the potential to be
useful to a lot of people.

I'd like to say that I'm now firmly convinced that the best place
to put this code would be on *top* of FreeType 2, in an independent
library. I'm not going to list the benefits of this approach here,
because clearly nobody has the time to perform this kind of work
(heck, I'm nearly retired from FreeType development at the moment).
We don't live in an ideal world, and it would be _stupid_ not to use
this code with such a feeble excuse.

I'm thus willing to accept the inclusion of this code within
FreeType 2. I just created a branch named "LAYOUT" on the repository,
and will update it with the most recent sources soon. By the way,
could Masatake send me privately a login and crypted password so
that I can give him full CVS write access if he wants to ?

The reason for the branch is that the API is clearly not finalized
(as explained below), and to allow parallel development without
disturbing too much FreeType releases. I hate branches, but so
be it.

However, I'll still be very pedantic regarding the API provided
to applications to manage and use the layout tables. I hope that
some day I'll find the time to port the code to an upper layer,
and I'd like to make minimal changes to the API to ease porting
of applications & libraries, so anything that looks too specific
to FT2 internals in the API will pass through very deep study :-)

Here's a brief analysis that doesn't go into too much technical
details:


I. Code overview:

I'll try to describe the code in a rather high-level overview so
that everyone interested can easily understand what has been written
without delving into pages of sources and specifications. It's not
like OTL and AAT are trivial things after all. :-)

- the code provides three main public header files to use:

   * <freetype/ftlayout.h>
       - provides an API to create layout contexts called "FT_FeaturesRequest".
         These are managed like FT_Size, which means that: they are owned by
         FT_Face objects, you can create more than one, only one is "active" at
         the time, etc...

       - provides an API to retrieve ligature caret division points
         for any glyph in a FT_Face, if any

       - provides a simple "FT_Glyphs_Array" type, equivalent to a Java2D
         "glyph vector", used to perform glyph substitution (there is no
         support for positioning in the API for now)

   * <freetype/otlayout.h>

       - provides additionnal methods to use on a layout context corresponding
         to an OpenType Layout font file, through the "OTL_FeaturesRequest" 
type.
         It allows to scan the GSUB/GPOS tables for scripts, languages, 
features,
         as well as "record" a set of features to later apply to glyph arrays

   * <freetype/gxlayout.h>

       - provides additionnal methods to use on a layout context corresponding
         to a TrueType GX/AAT font file, through the "GX_FeaturesRequest" type.
         It allows to list the AAT features recognized by the font, as well as
         to specify the state/value of any feature setting the application
         wants.

         Note that "OTL features" and "AAT features" are very different animals,
         despite their names, and are used in dramatically distinct ways.

- the implementation is made of:

  * the file "src/base/ftlayout.c", used to implement the public API parts of
    <freetype/ftlayout.h> as well as a few helper functions used by the 
format-specific
    implementations described below.

  * the directory "src/otlayout", which contains the OTL_FeaturesRequest 
implementation
    as a FreeType service (i.e. an optional module). However, this directory 
also contains
    code to implement the public API in <freetype/otlayout.h>

  * similarly, "src/gxlayout" is used to implement the GX_FeaturesRequest 
functions as
    a service, as well as the public API of <freetype/gxlayout.h>

- generally speaking, tables are not parsed directly in memory. Instead, 
similarly to
  the OTL support in FreeType 1, the code does load all data into 
library-specific
  structures, and does all processing within these. This is however an 
implementation
  detail that is not revealed by the API, and could be changed (hopefully) in 
the
  future.

  However, it means that most of the sources are really made of structure 
definitions and
  table loading/freeing. Processing representing only a small fraction here. 
And memory
  usage is pretty much more important than really needed. Well, I already said 
it, we don't
  live in an ideal world.

  I'd like to remind you that all OTL and AAT tables are clearly designed in a 
way that
  parsing them directly in memory is infinitely easier. The hard part is 
validating the
  input to ensure that you will not jump to unkown areas of memory due to a bad 
offset.
  you clearly don't want to check every offset in the table during glyph 
processing :-)

II. API Comments:

* the API is still _clearly_ alpha/beta, since positionning support isn't there 
yet
  and will clearly require changes to public structures like FTL_GlyphRec to add
  positioning information; or maybe a second glyph record type like in FT1 ?

  I'm also unsure that the current API matches the requirement of a project 
like Pango,
  even in the case of substitution (I'm mainly thinking about the issue of 
logical clusters
  and re-ordering), though I'd like Owen's opinion on the subject.

* I see three problems with the FTL_FeaturesRequest class, and its children:

   - first, it is used to access font-specific information that does not
     depend on user choice (i.e. a "request"), like OTL script/languages
     lists or AAT features/settings info. I find this rather confusing for
     the typical user, even one that knowns the OTL & AAT specs well.

     I'd rather like to see two new objects, like "OTL_Info" and "GX_Info" to
     report this data, extracted from the FT_Face itself, and limit the 
"Request"
     objects to hold pure applicative choices. The "OTL_FeaturesRequest" would
     thus be equivalent to the Pango "ruleset"; This would also clarify the
     current mess used to "manage" GX/AAT features and settings (the current
     API makes it very hard to understand what is what).

   - I'd like the API names to be more consistent than the current state of
     the FreeType API. I strongly favor the "FTL_ObjectType_DoAction" naming
     convention for any new development, because "FT_Do_Object_Type_Action"
     can be more confusing to users. I would also suggest renaming classes
     to something more meaningful.

   - the "abstract" FTL_FeaturesRequest object is complete overkill. It's only
     use is to basically allow calling FTL_Substitute_Glyphs with a FT_Face
     parameter in a format-independent way.

     However, the differences between OTL and AAT are so drastic that I'd
     clearly prefer to see something like the following:

          OTL_Info     => "list" all OpenType Layout data for applications. One 
per FT_Face

          GX_Info      => "list" all TrueType GX/AAT data for applications. 
This really
                          means supported features and their default settings 
(for
                          exclusive ones)

          OTL_Ruleset  => used as a substitution/positioning features context 
for OTL.
                          zero or more, owned by the FT_Face and destroyed with 
it. No need
                          for an implicit "active" one.

            OTL_Ruleset_New( FT_Face       face,
                             OTL_Ruleset  *aruleset );

            OTL_Ruleset_Free( OTL_Ruleset  ruleset );

            OTL_Ruleset_ClearFeatures( OTL_Ruleset  ruleset );

            OTL_Ruleset_AddFeature( OTL_Ruleset    ruleset,
                                    OTL_TableType  table_type,
                                    OTL_Tag        feature,
                                    OTL_UInt       properties );

            OTL_Ruleset_SubstituteGlyphs( OTL_Ruleset     ruleset,
                                          FTL_GlyphArray  in,
                                          FTL_GlyphArray  out );


          GX_Settings  => used as a substitution/positioning settings context 
for GX/AAT.
                          zero or more, owned by the FT_Face and destroyed with 
it. No
                          need for an implicit "active" one.

            GX_Settings_New( FT_Face       face,
                             GX_Settings  *asettings );

            GX_Settings_Free( GX_Settings  settings );

            GX_Settings_ClearFeatures( GX_Settings  settings );

            GX_Settings_SetFeature( GX_Settings  settings,
                                    GX_Feature   feature,
                                    FL_UInt      value );

            GX_Settings_SubstituteGlyphs( GX_Settings     settings,
                                          FTL_GlyphArray  in,
                                          FTL_GlyphArray  out );

        I know that these two objects look like they have the same methods. 
However,
        they are used in very distinct ways (for example, you typically need to 
apply
        several ruleset substitutions in very script-specific order to perform 
correct
        OpenType Layout, while only one call to GX_Settings_SubstituteGlyphs is 
ever
        needed, independently of the script/language you're using)

        it may be possible to have "OTL_Ruleset" and "GX_Settings" derive from 
a common
        type in the _implementation_, but I don't think this is really useful 
in the API.

        change "FTL_Glyphs_Array" to "FTL_GlyphArray".



III. Implementation Comments:

  there's one thing I don't like in the current implementation: it's the fact 
that
  the public APIs of <freetype/gxlayout.h> and <freetype/otlayout.h> are 
implemented
  within the optional modules.

  they should be implemented in "src/base/gxlayout.c" and 
"src/base/otlayout.c", these
  sources acting as simple stubs to connect to the corresponding services, or 
returning
  an error code when these are not available.

  other than that, the code itself is very solid. There are countless little 
thwarts that
  can be (very easily) fixed though. For examples:

   * replace code like:
       if ( (error = FT_READ_SHORT(value) ) goto Exit;
     with:
       if ( FT_READ_SHORT(value) ) goto Exit;

      and should be replaced with the simpler:

   * all FT_Frame_Field arrays must be defined constant, with "static const 
FT_Frame_Field"

   * many other little things I don't remember well. But the CVS branch is here 
to fix all
     of this.

Again, I'd like to thank Masatake for his work. I'm impatiently waiting for 
your login/pass
for the CVS repository, as well as opinions of all interested developers on the 
list.

Best Regards,

- David Turner
- The FreeType Project  (www.freetype.org)