FileDocCategorySizeDatePackage
LetterTokenizer.javaAPI DocApache Lucene 1.4.31361Tue Mar 30 00:48:00 BST 2004org.apache.lucene.analysis

LetterTokenizer

public class LetterTokenizer extends CharTokenizer
A LetterTokenizer is a tokenizer that divides text at non-letters. That's to say, it defines tokens as maximal strings of adjacent letters, as defined by java.lang.Character.isLetter() predicate. Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

Fields Summary
Constructors Summary
public LetterTokenizer(Reader in)
Construct a new LetterTokenizer.

    super(in);
Methods Summary
protected booleanisTokenChar(char c)
Collects only characters which satisfy {@link Character#isLetter(char)}.

    return Character.isLetter(c);