bug#19993: 25.0.50; Unicode fonts defective on Windows

[Eli Zaretskii]

> Date: Wed, 11 Mar 2015 22:21:23 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 19993@debbugs.gnu.org
> 
> > Date: Wed, 11 Mar 2015 12:49:39 -0700
> > From: Ilya Zakharevich <ilya@math.berkeley.edu>
> > Cc: 19993@debbugs.gnu.org
> > 
> > So the real question is not whether the presence of a Subset is used
> > as filters, but: is
> > 
> >     the presence of the required character in the font
> > 
> > used as a filter.
> 
> When a font matches all the other constraints, then yes, it is
> actually tested for whether it supports the specific character we need
> to display.  See font_has_char and its callers.

I had a few minutes to spare, so I took a closer look at the code.

The problem with the Mathematical Alphanumeric Symbols block is much
more prosaic than you thought: fontset.el breaks this block into
several distinct pseudo-scripts (don't know why, perhaps for
compatibility with something on Unix), but no one has taught w32font.c
to do the same for the corresponding Unicode subrange.  So Emacs was
asking for, say, 'mathematical-italic' "script", but w32font.c was
comparing that with 'mathematical', and was rejecting the fonts that
support these characters.  This is now fixed in commit fc10058 on
master.  You should now be able to type "C-x 8 RET 1d400 RET" and see
the character displayed.

While at that, I also added the missing subranges that for some reason
unknown to me were commented out; for many of them, I could verify
that adding them makes the corresponding characters displayable by
default, where they previously weren't.  (I couldn't verify that for
some of the scripts for which I have no fonts.)  A few subranges were
left out, and I added comments explaining why.

With that out of our way, part of the problem is solved.  Part, but
not all of it.  Because it is true: Emacs searches the fonts installed
on the system mostly by requiring only that the font supports the
script to which the character belongs, but without opening the font
and checking whether the specific character we are about to display
has a glyph in the font.  Here's the crucial piece of code (from
fontset.c):

          /* Find a font best-matching with the spec without checking
             the support of the character C.  That checking is costly,
             and even without the checking, the found font supports C
             in high possibility.  */
          font_entity = font_find_for_lface (f, face->lface,
                                             FONT_DEF_SPEC (font_def), -1);

That -1 as the last argument tells font_find_for_lface to not check
support for the character.

So yes, if a font claims support for a script, but actually supports
very little of it, it is quite possible that Emacs will try to use it,
and will then be unable to display the missing characters.

I know about font search on Unix even less than I know for Windows, so
I cannot tell if on Unix we are smarter about this.  I see that
ftfont.c uses fontconfig functions to verify that the representative
character of the required script (set up on fontset.el) is part of the
charset supported by a font, but I don't know if that looks into the
font, and in any case we only have at most 1 representative character
for all but a few scripts.  So misses are still possible; or maybe I'm
missing something.

Assuming that we want to become smarter about this, we could do one or
both of the following:

  . have a database of fonts which are _not_ to be used for certain
    scripts, because it is known that their coverage is poor

  . have a more elaborate default fontset that favors specific fonts
    for scripts which these fonts are known to support well

One problem with both of these is that it's hard to recommend fonts
because many good fonts are non-free.

If it turns out that these problems are Windows-specific, the above
can be done for Windows only (like w32-standard-fontset-spec).