FileDocCategorySizeDatePackage
LowerCaseTokenizer.javaAPI DocApache Lucene 1.4.31553Tue Mar 30 00:48:00 BST 2004org.apache.lucene.analysis

LowerCaseTokenizer

public final class LowerCaseTokenizer extends LetterTokenizer
LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of LetterTokenizer and LowerCaseFilter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.

Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

Fields Summary
Constructors Summary
public LowerCaseTokenizer(Reader in)
Construct a new LowerCaseTokenizer.

    super(in);
  
Methods Summary
protected charnormalize(char c)
Collects only characters which satisfy {@link Character#isLetter(char)}.

    return Character.toLowerCase(c);