Re: font substitution

[Yen-Ju Chen]

On 8/12/07, Fred Kiefer <fredkiefer@gmx.de> wrote:
> I tried both the AR PL fonts and they seem to work for me. The problem
> you reported in the other mail was when creating the character set, not
> when checking if a character was included. Perhaps you could send me
> your test file. At the moment I am not able to reproduce the problem.
>
> Excluding some fonts from the check wont be an option as people may want
> to use this fonts anyway and then we have the same problem. We really
> need to find out, what is going wrong here.
>
> And GNUstep should be able to support all Unicode characters, if not we
> need to change this. From looking at the code I see no limitation.

  Here is the text I used. It is in UTF-8 encoding.
  You need a Chinese font (AR PL...)
  and another font which has better coverage than usual,
  probably one of the DejaVu font for a row of symbol in the bottom.

  Yen-J

>
> Cheers,
> Fred
>
> Yen-Ju Chen wrote:
> > On 8/11/07, Fred Kiefer <fredkiefer@gmx.de> wrote:
> >> Thank you for the patches. I already applied the first one. For the
> >> other two I would like to understand the problem they try to solve a bit
> >> better, before working around it.
> >>
> >> The illegal value for the character set are really strange. The value
> >> GNUstep checks  against is 1114112, which should be the maximal allowed
> >> UNICODE value. I could understand, if GNUstep would be off by one. Could
> >> you please test this or send me such a font, then I would have a look
> >> myself.
> >
> >   I use the font here:
> >   http://download.gna.org//etoile/etoile-default-fonts.tar.gz
> >   One of the Chinese font (AR PL...) is broken.
> >   Actually there are two possibility.
> >   NSCharacterSet support longCharacterIsMember:,
> >   which take 32-bit integer instead of unichar (16-bit).
> >   It indicates that unicode can go beyond 16-bit.
> >   If I use -longCharacterIsMember: to get font coverage,
> >   it complains something is too big.
> >   So I suspect either GNUstep cannot support 32-bit glyph,
> >   or the font has a wrong table inside.
> >   Actually it happens to me some time ago with another set of fonts.
> >   Considering the nature of open source,
> >   I am not surprised that some of the font has the wrong table inside.
> >   For art backend, we have limited nfont packages.
> >   So it is not a big problem to find the bad one and remove it.
> >   But for cairo backend, it gets all fonts from X, which is a LOT.
> >   What's the chance of any of them has broken table inside ?
> >   So the purpose of not using all availbe fonts is
> >   1. users can control which fonts to use and avoid bad font from system.
> >   2. GNUstep may not support for font beyond 16-bit glyph (maybe).
> >   And I really don't see what we can get from using all available
> > fonts for substitution.
> >
> >> The matrix swapping is also strange. In many case your work around would
> >> be fine, but in others when the matrix was explicitly set it would be
> >> wrong. Here it would be nice to know which case are handled correct and
> >> which wrong by the current code. Is there anything special in the way
> >> you created you font?
> >
> >   The font I use is pretty much the standard font you can get (see above 
> > link).
> >   While I don't understand font matrix that much,
> >   it does not make sense to me that you can apply a matrix of English font
> >   to Chinese font, not to mention other languages, like Indian or Arabic 
> > font.
> >   So sharing font matrix across fonts of different languages looks
> > more wrong to me.
> >   And if users set matrix explicitly to the default font,
> >   it means he want to change matrix of default font, not any other.
> >   Automatically applying such matrix to any other fonts also seems wrong to 
> > me.
> >
> >   Regards
> >
> >   Yen-Ju
> >
> >> Cheers,
> >> Fred
> >>
> >> Yen-Ju Chen wrote:
> >>> O.K. Sorry for another reply
> >>> This patch fixes the upside-down problem.
> >>> The attached screenshot uses English font as default font
> >>> and Chinese font as substitute.
> >>> I probably need to find a Japanese font and a symbol font to test
> >>> multiple substitute.
> >>> By the way, the speed is not bad.
> >>> Previously I said that it takes 1-2 seconds for 25,000+ Chinese glyphs
> >>> is including putting all glyphs on a NSMatrix.
> >>> So it is fast to get the coverage in NSFont.
> >>>
> >>> Yen-Ju
> >>>
> >>> On 8/10/07, Yen-Ju Chen <yjchenx@gmail.com> wrote:
> >>>> Another issue is that sometimes freetype fails to get the coverage.
> >>>> Therefore, NSCharacterSet raise an exception:
> >>>> "Specified range exceeds character set".
> >>>> So this patch prevents NSAttributedString from using all available fonts
> >>>> to eliminate the possibility of using bad fonts.
> >>>> It only uses user specified fonts for substitution.
> >>>> And I hope GNUstep can add an user default to specify the preferred 
> >>>> fonts.
> >>>>
> >>>> Once the patch is applied, the font substitute basically works, but
> >>>> has one problem.
> >>>> I attach a screenshot to explain that.
> >>>> The left window is using English font as default font and no preferred 
> >>>> font.
> >>>> So you only see English 'GNUstep'.
> >>>> The middle window is using Chinese font as default font and no preferred 
> >>>> font.
> >>>> Because Chinese font also contains English letter,
> >>>> you see both English and Chinese.
> >>>> You can tell the English letter is from different font.
> >>>> The right window is using English font as default font and Chinese
> >>>> font as substitute.
> >>>> Again, you can see both English and Chinese character.
> >>>> But the funny thing is the Chinese character is upside-down.
> >>>> I have no idea why it happens, but I suspect gui or back cache the font 
> >>>> matrix
> >>>> so the matrix of English font is applied to the Chinese font.
> >>>> The hint of Chinese font is also different between middle and right 
> >>>> window.
> >>>> The middle one is sharp while the right one is blur.
> >>>> It indicates something is cached, probably for the English font at the
> >>>> start of the text.
> >>>> Maybe someone has insights about what may cause it.
> >>>>
> >>>> Yen-Ju
> >>>>
> >>>> On 8/10/07, Yen-Ju Chen <yjchenx@gmail.com> wrote:
> >>>>> Hi,
> >>>>>
> >>>>>   [NSFont coveredCharacterSet] is broken.
> >>>>>   I attach a patch.
> >>>>>   A quick test shows that a Chinese font which contains 25,000+ glyph
> >>>>>   takes only about 1-2 seconds to go through.
> >>>>>   It's on a powerbook G4 1GHz.
> >>>>>   So cache may not be needed for coveredCharacterSet.
> >>>>>   I haven't test the preferredFont yet.
> >>>>>   Hope this patch can go in soon.
> >>>>>
> >>>>>   Yen-Ju
> >>>>>
> >>>>> On 8/9/07, Yen-Ju Chen <yjchenx@gmail.com> wrote:
> >>>>>> On 8/9/07, Fred Kiefer <fredkiefer@gmx.de> wrote:
> >>>>>>> Yen-Ju Chen wrote:
> >>>>>>>> On 8/8/07, Fred Kiefer <fredkiefer@gmx.de> wrote:
> >>>>>>>>> Yen-Ju Chen wrote:
> >>>>>>>>>>   Another thing I am thinking is to save the coveredCharacterSet 
> >>>>>>>>>> on disk.
> >>>>>>>>>>   Font does not change frequently,
> >>>>>>>>>>   and it does take time to generate coveredCharacterSet for each 
> >>>>>>>>>> font,
> >>>>>>>>>>   especially for CJK.
> >>>>>>>>>>   If the cached coveredCharacterSet can be saved on disk and 
> >>>>>>>>>> loaded later,
> >>>>>>>>>>   it can reduce the overhead of starting an application.
> >>>>>>>>>>   For art backend, it can even pack the saved coveredCharacterSet 
> >>>>>>>>>> in nfont.
> >>>>>>>>>>   So for small font, it is fine to check the covered character 
> >>>>>>>>>> each time.
> >>>>>>>>>>   For big font, saved character set is better.
> >>>>>>>>>>
> >>>>>>>>> Great idea! Would you like to work on that?
> >>>>>>>>   If the patch you have goes into the -trunk,
> >>>>>>>>   I can play with it and see how much it effects the speed
> >>>>>>>>   and decide what's the best way to do the cache if needed.
> >>>>>>>>
> >>>>>>> I just committed this change, now it is your turn :-)
> >>>>>>   Thanx. I will take a look shortly.
> >>>>>>
> >>>>>>   Yen-Ju
> >>>>>>
> >>>>
> >>>> ------------------------------------------------------------------------
> >>>>
> >>>> Index: NSAttributedString.m
> >>>> ===================================================================
> >>>> --- NSAttributedString.m     (revision 25384)
> >>>> +++ NSAttributedString.m     (working copy)
> >>>> @@ -990,7 +990,7 @@
> >>>>  - (NSFont*)_substituteFontWithName: (NSString*)fontName font: 
> >>>> (NSFont*)baseFont
> >>>>  {
> >>>>    // FIXME: Catch case were baseFont is nil
> >>>> -  return [NSFont fontWithName: fontName matrix: [baseFont matrix]];
> >>>> +  return [NSFont fontWithName: fontName size: [baseFont pointSize]];
> >>>>  }
> >>>>
> >>>>  - (NSFont*)_substituteFontFor: (unichar)uchar font: (NSFont*)baseFont 
> >>>> fromList: (NSArray *)fonts
> >>>> @@ -1061,13 +1061,6 @@
> >>>>      {
> >>>>        return subFont;
> >>>>      }
> >>>> -
> >>>> -  subFont = [self _substituteFontFor: uchar font: baseFont fromList:
> >>>> -                      [[NSFontManager sharedFontManager] 
> >>>> availableFonts]];
> >>>> -  if (subFont != nil)
> >>>> -    {
> >>>> -      return subFont;
> >>>> -    }
> >>>>
> >>>>    return nil;
> >>>>  }
> >>>>
> >>>> ------------------------------------------------------------------------
> >>>>
> >>
> >
>
>

GNUstep.txt
Description: Text document