Re: font substitution

[Fred Kiefer]

I tried both the AR PL fonts and they seem to work for me. The problem
you reported in the other mail was when creating the character set, not
when checking if a character was included. Perhaps you could send me
your test file. At the moment I am not able to reproduce the problem.

Excluding some fonts from the check wont be an option as people may want
to use this fonts anyway and then we have the same problem. We really
need to find out, what is going wrong here.

And GNUstep should be able to support all Unicode characters, if not we
need to change this. From looking at the code I see no limitation.

Cheers,
Fred

Yen-Ju Chen wrote:
> On 8/11/07, Fred Kiefer <fredkiefer@gmx.de> wrote:
>> Thank you for the patches. I already applied the first one. For the
>> other two I would like to understand the problem they try to solve a bit
>> better, before working around it.
>>
>> The illegal value for the character set are really strange. The value
>> GNUstep checks  against is 1114112, which should be the maximal allowed
>> UNICODE value. I could understand, if GNUstep would be off by one. Could
>> you please test this or send me such a font, then I would have a look
>> myself.
> 
>   I use the font here:
>   http://download.gna.org//etoile/etoile-default-fonts.tar.gz
>   One of the Chinese font (AR PL...) is broken.
>   Actually there are two possibility.
>   NSCharacterSet support longCharacterIsMember:,
>   which take 32-bit integer instead of unichar (16-bit).
>   It indicates that unicode can go beyond 16-bit.
>   If I use -longCharacterIsMember: to get font coverage,
>   it complains something is too big.
>   So I suspect either GNUstep cannot support 32-bit glyph,
>   or the font has a wrong table inside.
>   Actually it happens to me some time ago with another set of fonts.
>   Considering the nature of open source,
>   I am not surprised that some of the font has the wrong table inside.
>   For art backend, we have limited nfont packages.
>   So it is not a big problem to find the bad one and remove it.
>   But for cairo backend, it gets all fonts from X, which is a LOT.
>   What's the chance of any of them has broken table inside ?
>   So the purpose of not using all availbe fonts is
>   1. users can control which fonts to use and avoid bad font from system.
>   2. GNUstep may not support for font beyond 16-bit glyph (maybe).
>   And I really don't see what we can get from using all available
> fonts for substitution.
> 
>> The matrix swapping is also strange. In many case your work around would
>> be fine, but in others when the matrix was explicitly set it would be
>> wrong. Here it would be nice to know which case are handled correct and
>> which wrong by the current code. Is there anything special in the way
>> you created you font?
> 
>   The font I use is pretty much the standard font you can get (see above 
> link).
>   While I don't understand font matrix that much,
>   it does not make sense to me that you can apply a matrix of English font
>   to Chinese font, not to mention other languages, like Indian or Arabic font.
>   So sharing font matrix across fonts of different languages looks
> more wrong to me.
>   And if users set matrix explicitly to the default font,
>   it means he want to change matrix of default font, not any other.
>   Automatically applying such matrix to any other fonts also seems wrong to 
> me.
> 
>   Regards
> 
>   Yen-Ju
> 
>> Cheers,
>> Fred
>>
>> Yen-Ju Chen wrote:
>>> O.K. Sorry for another reply
>>> This patch fixes the upside-down problem.
>>> The attached screenshot uses English font as default font
>>> and Chinese font as substitute.
>>> I probably need to find a Japanese font and a symbol font to test
>>> multiple substitute.
>>> By the way, the speed is not bad.
>>> Previously I said that it takes 1-2 seconds for 25,000+ Chinese glyphs
>>> is including putting all glyphs on a NSMatrix.
>>> So it is fast to get the coverage in NSFont.
>>>
>>> Yen-Ju
>>>
>>> On 8/10/07, Yen-Ju Chen <yjchenx@gmail.com> wrote:
>>>> Another issue is that sometimes freetype fails to get the coverage.
>>>> Therefore, NSCharacterSet raise an exception:
>>>> "Specified range exceeds character set".
>>>> So this patch prevents NSAttributedString from using all available fonts
>>>> to eliminate the possibility of using bad fonts.
>>>> It only uses user specified fonts for substitution.
>>>> And I hope GNUstep can add an user default to specify the preferred fonts.
>>>>
>>>> Once the patch is applied, the font substitute basically works, but
>>>> has one problem.
>>>> I attach a screenshot to explain that.
>>>> The left window is using English font as default font and no preferred 
>>>> font.
>>>> So you only see English 'GNUstep'.
>>>> The middle window is using Chinese font as default font and no preferred 
>>>> font.
>>>> Because Chinese font also contains English letter,
>>>> you see both English and Chinese.
>>>> You can tell the English letter is from different font.
>>>> The right window is using English font as default font and Chinese
>>>> font as substitute.
>>>> Again, you can see both English and Chinese character.
>>>> But the funny thing is the Chinese character is upside-down.
>>>> I have no idea why it happens, but I suspect gui or back cache the font 
>>>> matrix
>>>> so the matrix of English font is applied to the Chinese font.
>>>> The hint of Chinese font is also different between middle and right window.
>>>> The middle one is sharp while the right one is blur.
>>>> It indicates something is cached, probably for the English font at the
>>>> start of the text.
>>>> Maybe someone has insights about what may cause it.
>>>>
>>>> Yen-Ju
>>>>
>>>> On 8/10/07, Yen-Ju Chen <yjchenx@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>>   [NSFont coveredCharacterSet] is broken.
>>>>>   I attach a patch.
>>>>>   A quick test shows that a Chinese font which contains 25,000+ glyph
>>>>>   takes only about 1-2 seconds to go through.
>>>>>   It's on a powerbook G4 1GHz.
>>>>>   So cache may not be needed for coveredCharacterSet.
>>>>>   I haven't test the preferredFont yet.
>>>>>   Hope this patch can go in soon.
>>>>>
>>>>>   Yen-Ju
>>>>>
>>>>> On 8/9/07, Yen-Ju Chen <yjchenx@gmail.com> wrote:
>>>>>> On 8/9/07, Fred Kiefer <fredkiefer@gmx.de> wrote:
>>>>>>> Yen-Ju Chen wrote:
>>>>>>>> On 8/8/07, Fred Kiefer <fredkiefer@gmx.de> wrote:
>>>>>>>>> Yen-Ju Chen wrote:
>>>>>>>>>>   Another thing I am thinking is to save the coveredCharacterSet on 
>>>>>>>>>> disk.
>>>>>>>>>>   Font does not change frequently,
>>>>>>>>>>   and it does take time to generate coveredCharacterSet for each 
>>>>>>>>>> font,
>>>>>>>>>>   especially for CJK.
>>>>>>>>>>   If the cached coveredCharacterSet can be saved on disk and loaded 
>>>>>>>>>> later,
>>>>>>>>>>   it can reduce the overhead of starting an application.
>>>>>>>>>>   For art backend, it can even pack the saved coveredCharacterSet in 
>>>>>>>>>> nfont.
>>>>>>>>>>   So for small font, it is fine to check the covered character each 
>>>>>>>>>> time.
>>>>>>>>>>   For big font, saved character set is better.
>>>>>>>>>>
>>>>>>>>> Great idea! Would you like to work on that?
>>>>>>>>   If the patch you have goes into the -trunk,
>>>>>>>>   I can play with it and see how much it effects the speed
>>>>>>>>   and decide what's the best way to do the cache if needed.
>>>>>>>>
>>>>>>> I just committed this change, now it is your turn :-)
>>>>>>   Thanx. I will take a look shortly.
>>>>>>
>>>>>>   Yen-Ju
>>>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> Index: NSAttributedString.m
>>>> ===================================================================
>>>> --- NSAttributedString.m     (revision 25384)
>>>> +++ NSAttributedString.m     (working copy)
>>>> @@ -990,7 +990,7 @@
>>>>  - (NSFont*)_substituteFontWithName: (NSString*)fontName font: 
>>>> (NSFont*)baseFont
>>>>  {
>>>>    // FIXME: Catch case were baseFont is nil
>>>> -  return [NSFont fontWithName: fontName matrix: [baseFont matrix]];
>>>> +  return [NSFont fontWithName: fontName size: [baseFont pointSize]];
>>>>  }
>>>>
>>>>  - (NSFont*)_substituteFontFor: (unichar)uchar font: (NSFont*)baseFont 
>>>> fromList: (NSArray *)fonts
>>>> @@ -1061,13 +1061,6 @@
>>>>      {
>>>>        return subFont;
>>>>      }
>>>> -
>>>> -  subFont = [self _substituteFontFor: uchar font: baseFont fromList:
>>>> -                      [[NSFontManager sharedFontManager] availableFonts]];
>>>> -  if (subFont != nil)
>>>> -    {
>>>> -      return subFont;
>>>> -    }
>>>>
>>>>    return nil;
>>>>  }
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>
>