[Devel] pshalgo3.c has an access violation in low memory conditions

[Graham Asher]

Dear FreeTypers,

I discovered this defect while working on the Symbian OS port of CartoType,
a new cartography engine that uses FreeType for all its text. The Symbian OS
has a handy feature that allows you to simulate heap failure, so that you
can test the robustness of an application under low memory conditions. While
doing this my app crashed with an access violation in FT_Free caused by the
argument "memory" being null.

What happens is that if psh3_glyph_init runs out of memory when called by
ps3_hints_apply, it is possible for glyph->memory to be null. In the
following code (the first part of psh3_glyph_init), if the second of the
calls to FT_NEW_ARRAY fails, glyph->memory will remain null but the first
array will still need to be deleted:

  static FT_Error
  psh3_glyph_init( PSH3_Glyph   glyph,
                   FT_Outline*  outline,
                   PS_Hints     ps_hints,
                   PSH_Globals  globals )
  {
    FT_Error   error;
    FT_Memory  memory;


    /* clear all fields */
    FT_MEM_ZERO( glyph, sizeof ( *glyph ) );

    memory = globals->memory;

    /* allocate and setup points + contours arrays */
    if ( FT_NEW_ARRAY( glyph->points,   outline->n_points   ) ||
         FT_NEW_ARRAY( glyph->contours, outline->n_contours ) )
      goto Exit;

The fix is to insert the line marked with /* NEW LINE */ below (and remove
the identical line from lower down in the function):

  static FT_Error
  psh3_glyph_init( PSH3_Glyph   glyph,
                   FT_Outline*  outline,
                   PS_Hints     ps_hints,
                   PSH_Globals  globals )
  {
    FT_Error   error;
    FT_Memory  memory;


    /* clear all fields */
    FT_MEM_ZERO( glyph, sizeof ( *glyph ) );

    memory = globals->memory;
    glyph->memory = memory; /* NEW LINE */

    /* allocate and setup points + contours arrays */
    if ( FT_NEW_ARRAY( glyph->points,   outline->n_points   ) ||
         FT_NEW_ARRAY( glyph->contours, outline->n_contours ) )
      goto Exit;

There are almost certainly other potential problems with low memory in
FreeType and I will report any that I find, but I must urge other people to
look as well. Symbian OS developers have it easy, because they can just
press Ctrl+Alt+Shift+P while the app is running and turn on simulated heap
failure.

Best wishes,

Graham Asher