Home » lucene-2.3.2-src » org.apache » lucene » analysis » [javadoc | source]

org.apache.lucene.analysis
public class: LetterTokenizer [javadoc | source]

java.lang.Object
   org.apache.lucene.analysis.TokenStream
      org.apache.lucene.analysis.Tokenizer
         org.apache.lucene.analysis.CharTokenizer
            org.apache.lucene.analysis.LetterTokenizer

Direct Known Subclasses:
    LowerCaseTokenizer

A LetterTokenizer is a tokenizer that divides text at non-letters. That's to say, it defines tokens as maximal strings of adjacent letters, as defined by java.lang.Character.isLetter() predicate. Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

Fields inherited from org.apache.lucene.analysis.Tokenizer:
input

Constructor:
public LetterTokenizer(Reader in) { super(in); } Construct a new LetterTokenizer.

Method from org.apache.lucene.analysis.LetterTokenizer Summary:
isTokenChar

Methods from org.apache.lucene.analysis.CharTokenizer:
isTokenChar,   next,   normalize,   reset

Methods from org.apache.lucene.analysis.Tokenizer:
close,   reset

Methods from org.apache.lucene.analysis.TokenStream:
close,   next,   next,   reset

Methods from java.lang.Object:
equals,   getClass,   hashCode,   notify,   notifyAll,   toString,   wait,   wait,   wait

Method from org.apache.lucene.analysis.LetterTokenizer Detail:
protected boolean isTokenChar(char c) { return Character.isLetter(c); } Collects only characters which satisfy Character#isLetter(char) .