[freetype2] master a4c8f21 1/2: [base] Reject combinations of incompatible `FT_OPEN_XXX` flags.

[Werner Lemberg]

branch: master
commit a4c8f21ae7938a99fbb556fccfd0da3457d93643
Author: Oleg Oshmyan <chortos@inbox.lv>
Commit: Werner Lemberg <wl@gnu.org>

    [base] Reject combinations of incompatible `FT_OPEN_XXX` flags.
    
    The three modes are mutually exclusive, and the documentation of the
    `FT_OPEN_XXX` constants notes this.  However, there was no check to
    validate this in the code, and the documentation on `FT_Open_Args`
    claimed that the corresponding bits were checked in a well-defined
    order, implying it was valid (if useless) to specify more than one.
    Ironically, this documented order did not agree with the actual
    code, so it could not be relied upon; hopefully, nobody did this and
    nobody will be hurt by the new validation.
    
    Even if multiple mode bits were allowed, they could cause memory
    leaks: if both `FT_OPEN_STREAM` and `stream` are set along with
    either `FT_OPEN_MEMORY` or `FT_OPEN_PATHNAME`, then `FT_Stream_New`
    allocated a new stream but `FT_Open_Face` marked it as an 'external'
    stream, so the stream object was never released.
    
    * src/base/ftobjs.c (FT_Stream_New): Reject incompatible
    `FT_OPEN_XXX` flags.
---
 ChangeLog                   | 22 ++++++++++++++++++++++
 include/freetype/freetype.h |  6 ++++--
 src/base/ftobjs.c           |  9 ++++++---
 3 files changed, 32 insertions(+), 5 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index d8ef3e2..5a40b09 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,25 @@
+2021-07-13  Oleg Oshmyan  <chortos@inbox.lv>
+
+       [base] Reject combinations of incompatible `FT_OPEN_XXX` flags.
+
+       The three modes are mutually exclusive, and the documentation of the
+       `FT_OPEN_XXX` constants notes this.  However, there was no check to
+       validate this in the code, and the documentation on `FT_Open_Args`
+       claimed that the corresponding bits were checked in a well-defined
+       order, implying it was valid (if useless) to specify more than one.
+       Ironically, this documented order did not agree with the actual
+       code, so it could not be relied upon; hopefully, nobody did this and
+       nobody will be hurt by the new validation.
+
+       Even if multiple mode bits were allowed, they could cause memory
+       leaks: if both `FT_OPEN_STREAM` and `stream` are set along with
+       either `FT_OPEN_MEMORY` or `FT_OPEN_PATHNAME`, then `FT_Stream_New`
+       allocated a new stream but `FT_Open_Face` marked it as an 'external'
+       stream, so the stream object was never released.
+
+       * src/base/ftobjs.c (FT_Stream_New): Reject incompatible
+       `FT_OPEN_XXX` flags.
+
 2021-07-12  Alex Richardson  <Alexander.Richardson@cl.cam.ac.uk>
 
        * meson.build: Fix build for other UNIX systems (e.g., FreeBSD).
diff --git a/include/freetype/freetype.h b/include/freetype/freetype.h
index f393e3d..598abd8 100644
--- a/include/freetype/freetype.h
+++ b/include/freetype/freetype.h
@@ -2113,8 +2113,7 @@ FT_BEGIN_HEADER
    *     Extra parameters passed to the font driver when opening a new face.
    *
    * @note:
-   *   The stream type is determined by the contents of `flags` that are
-   *   tested in the following order by @FT_Open_Face:
+   *   The stream type is determined by the contents of `flags`:
    *
    *   If the @FT_OPEN_MEMORY bit is set, assume that this is a memory file
    *   of `memory_size` bytes, located at `memory_address`.  The data are not
@@ -2127,6 +2126,9 @@ FT_BEGIN_HEADER
    *   Otherwise, if the @FT_OPEN_PATHNAME bit is set, assume that this is a
    *   normal file and use `pathname` to open it.
    *
+   *   If none of the above bits are set or if multiple are set at the same
+   *   time, the flags are invalid and @FT_Open_Face fails.
+   *
    *   If the @FT_OPEN_DRIVER bit is set, @FT_Open_Face only tries to open
    *   the file with the driver whose handler is in `driver`.
    *
diff --git a/src/base/ftobjs.c b/src/base/ftobjs.c
index 4a80145..f7b2b3f 100644
--- a/src/base/ftobjs.c
+++ b/src/base/ftobjs.c
@@ -197,6 +197,7 @@
     FT_Error   error;
     FT_Memory  memory;
     FT_Stream  stream = NULL;
+    FT_UInt    mode;
 
 
     *astream = NULL;
@@ -208,13 +209,15 @@
       return FT_THROW( Invalid_Argument );
 
     memory = library->memory;
+    mode   = args->flags &
+               ( FT_OPEN_MEMORY | FT_OPEN_STREAM | FT_OPEN_PATHNAME );
 
     if ( FT_NEW( stream ) )
       goto Exit;
 
     stream->memory = memory;
 
-    if ( args->flags & FT_OPEN_MEMORY )
+    if ( mode == FT_OPEN_MEMORY )
     {
       /* create a memory-based stream */
       FT_Stream_OpenMemory( stream,
@@ -224,13 +227,13 @@
 
 #ifndef FT_CONFIG_OPTION_DISABLE_STREAM_SUPPORT
 
-    else if ( args->flags & FT_OPEN_PATHNAME )
+    else if ( mode == FT_OPEN_PATHNAME )
     {
       /* create a normal system stream */
       error = FT_Stream_Open( stream, args->pathname );
       stream->pathname.pointer = args->pathname;
     }
-    else if ( ( args->flags & FT_OPEN_STREAM ) && args->stream )
+    else if ( ( mode == FT_OPEN_STREAM ) && args->stream )
     {
       /* use an existing, user-provided stream */