Re: [ft-devel] Known problem with FT_StreamRec::base being non-NULL also for file-based streams?

[mpsuzuki]

On Wed, 10 Mar 2010 03:50:17 -0500
Behdad Esfahbod <address@hidden> wrote:
>Well, in short, all the hb_blob_t in HarfBuzz is about communicating to
>harfbuzz what it can do with the memory backing the font file.  There are
>three different cases we are interested in:
>
>  - The memory is read-only; harfbuzz will make a copy if it needs to modify 
> it.
>
>  - The memory is writable and it is ok to write to it.  harfbuzz will not
>make a copy.
>
>  - The memory is read-only, but can be made writable using mprotect() or
>similar (win32, ...) functionality.

>Currently the hb-ft glue layer assumes that font data is mmap()ed or are
>otherwise mprotect()able.

Thanks. I understand the info needed by HB is whether HB can
modify the memory image without duplication, or HB should copy
the memory image before duplication.

Just I've posted a proof of concept to know who allocated the
buffer in FT_Stream, but it is not best solution for this task.

I and Werner agreed that the easiest way to guarantee the
origin of buffer as mmap()ed or malloc()ed is the font
image preparation in HB/Pango side. But, taking a glance
on Pango, I guess, there might be some delay between the
invocation of FT_New_Face() and HB blob creation.
The duplication of unwritable font image to writable buffer
occurs for all faces? Or, the duplication occurs when the
first modification is tried (to fix OpenType bug in runtime)?

If latter scenario is correct - when Pango is going to
create FT_Face object, Pango cannot know if the duplication
will occur in future, so, my proposal (HB/Pango side font
image preparation) will cause unwanted memory consumation
by loading all faces to writable memory. It won't be good
idea.

# In Pango library, when PangoFT2Font->face is created once,
# it should not be changed anymore? If replacing the face is
# permitted, I want to create the earliest unwritable face
# from mmap()ed image, then replace it by writable face with
# malloc()-and-read() image when Pango/HB tries to modify it.
# Pango has an API to expose PangoFT2Font->face to the client
# (pango_ft2_font_get_face()), but it is classified as deprecated
# interface. I wish Pango library is changing to hide raw
# FT_Face object from Pango client.

> This fails for examples when:
>
>  - Font data is in ROM.  In this case mprotect() will fail and harfbuzz will
>make a copy of the memory.  Not a huge problem.

Indeed. If FT2 could mmap() readonly font file successfully,
mprotect() will fail.

>  - FreeType malloc()ed the font data.  In this case, mprotect() is not
>necessary and will probably affect memory beyond the font data (since mprotect
>works on whole pages).

Umm, I think, mprotect() for malloc()ed memory causes
undefined result.

http://www.opengroup.org/onlinepubs/000095399/functions/mprotect.html
>The behavior of this function is unspecified
>if the mapping was not established by a call to mmap().

To avoid such ambiguity, we should know if the buffer
is mmap()ed or malloc()ed, before mprotect() - am I
misunderstanding?

>  - Font data is coming from the user.  In this case it may not be desirable
>to modify the data.

Indeed. Could you tell me which function is used to push
user-provided font data? Is it in cairo layer?

>Adding API to FTStream to be able to detect the above cases, specially the
>user-provided data, would be useful.

Again thank you for comment about the additon of new API.

As I've sketched, it is possible to get detailed info of
FT_Stream object. My current sketch is huge, and I have
a few issues to be discussed for further improvement of
FT_Stream object.

Another idea is an addition of the arguments for FT_Open_Face(),
to specify 3 scenarios for font loading.

1) only mmap() is tried.
2) only malloc() + read() is tried.
3) mmap() is tried, then, malloc() + read() is tried (current behaviour)

By using 1) and 2), HB/Pango can distinguish the buffer is
mmap()ed or malloc()ed exactly. I will post a patch for FT2
and Pango for further discussion. Please wait a few days...

Regards,
mpsuzuki