Re: [Freecats-Dev] Translation Units indexing - a first draft

[David N. Welton]

"Thierry Sourbier" <address@hidden> writes:

> 1) Unicode character properties should be used to determine if a
> character is a letter/digit/punctuation mark.

Tcl deals with this out of the box, in theory:

string is punct $foo

        punct     Any Unicode punctuation character.

> 2) Word breaking is a challenging problem (think Japanese, Thai...).

It also has a 'string wordend'.

Don't know if these do what they should for Asian character sets,
although my inclination on any open source project is to get it
running first.  Maybe that means getting the project started with
European languages, and then mixing in others.  This has the
disadvantage that you might have to rework things later, but at least
you get something people can use and then they get interested in your
project...

> 4) If you want the TM to scale only store in the index the smallest
> "sub-words" (N-Gram in the litterature)

This is not really my department:-)

-- 
David N. Welton
   Consulting: http://www.dedasys.com/
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/