FileDocCategorySizeDatePackage
LetterTokenizer.javaAPI DocApache Lucene 2.1.01549Wed Feb 14 10:46:38 GMT 2007org.apache.lucene.analysis

LetterTokenizer

public class LetterTokenizer extends CharTokenizer
A LetterTokenizer is a tokenizer that divides text at non-letters. That's to say, it defines tokens as maximal strings of adjacent letters, as defined by java.lang.Character.isLetter() predicate. Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

Fields Summary
Constructors Summary
public LetterTokenizer(Reader in)
Construct a new LetterTokenizer.

    super(in);
Methods Summary
protected booleanisTokenChar(char c)
Collects only characters which satisfy {@link Character#isLetter(char)}.

    return Character.isLetter(c);