org.apache.lucene.index.memory: public class: SynonymTokenFilter

org.apache.lucene.index.memory
public class: SynonymTokenFilter [javadoc | source]

java.lang.Object
   org.apache.lucene.analysis.TokenStream
      org.apache.lucene.analysis.TokenFilter
         org.apache.lucene.index.memory.SynonymTokenFilter

Injects additional tokens for synonyms of token terms fetched from the underlying child stream; the child stream must deliver lowercase tokens for synonyms to be found.

author: whoschek.AT.lbl.DOT.gov -

Field Summary
public static final String	SYNONYM_TOKEN_TYPE	The Token.type used to indicate a synonym to higher level filters.

Fields inherited from org.apache.lucene.analysis.TokenFilter:
input

Constructor:

Constructor:
public SynonymTokenFilter(TokenStream input, SynonymMap synonyms, int maxSynonyms) { super(input); if (input == null) throw new IllegalArgumentException("input must not be null"); if (synonyms == null) throw new IllegalArgumentException("synonyms must not be null"); if (maxSynonyms < 0) throw new IllegalArgumentException("maxSynonyms must not be negative"); this.synonyms = synonyms; this.maxSynonyms = maxSynonyms; } Creates an instance for the given underlying stream and synonym table. Parameters: `input` - the underlying child token stream `synonyms` - the map used to extract synonyms for terms `maxSynonyms` - the maximum number of synonym tokens to return per underlying token word (a value of Integer.MAX_VALUE indicates unlimited)

 public SynonymTokenFilter(TokenStream input,
    SynonymMap synonyms,
    int maxSynonyms) {
  
    super(input);
    if (input == null)
      throw new IllegalArgumentException("input must not be null");
    if (synonyms == null)
      throw new IllegalArgumentException("synonyms must not be null");
    if (maxSynonyms <  0) 
      throw new IllegalArgumentException("maxSynonyms must not be negative");
    this.synonyms = synonyms;
    this.maxSynonyms = maxSynonyms;
}

Creates an instance for the given underlying stream and synonym table.

Parameters:

input - the underlying child token stream

synonyms - the map used to extract synonyms for terms

maxSynonyms - the maximum number of synonym tokens to return per underlying token word (a value of Integer.MAX_VALUE indicates unlimited)

Method from org.apache.lucene.index.memory.SynonymTokenFilter Summary:
createToken, next

Methods from org.apache.lucene.analysis.TokenFilter:
close

Methods from org.apache.lucene.analysis.TokenStream:
close, next, next, reset

Methods from java.lang.Object:
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method from org.apache.lucene.index.memory.SynonymTokenFilter Detail:
protected Token createToken(String synonym, Token current) { Token token = new Token( synonym, current.startOffset(), current.endOffset(), SYNONYM_TOKEN_TYPE); token.setPositionIncrement(0); return token; } Creates and returns a token for the given synonym of the current input token; Override for custom (stateless or stateful) behaviour, if desired.
public Token next() throws IOException { Token token; while (todo > 0 && index < stack.length) { // pop from stack token = createToken(stack[index++], current); if (token != null) { todo--; return token; } } token = input.next(); if (token == null) return null; // EOS; iterator exhausted stack = synonyms.getSynonyms(token.termText()); // push onto stack if (stack.length > maxSynonyms) randomize(stack); index = 0; current = token; todo = maxSynonyms; return token; } Returns the next token in the stream, or null at EOS.

Method from org.apache.lucene.index.memory.SynonymTokenFilter Detail:

 protected Token createToken(String synonym,
    Token current) {
    Token token = new Token(
      synonym, current.startOffset(), current.endOffset(), SYNONYM_TOKEN_TYPE);
    token.setPositionIncrement(0);
    return token;
}

Creates and returns a token for the given synonym of the current input token; Override for custom (stateless or stateful) behaviour, if desired.

 public Token next() throws IOException {
    Token token;
    while (todo  > 0 && index <  stack.length) { // pop from stack
      token = createToken(stack[index++], current);
      if (token != null) {
        todo--;
        return token;
      }
    }
    token = input.next();
    if (token == null) return null; // EOS; iterator exhausted
    stack = synonyms.getSynonyms(token.termText()); // push onto stack
    if (stack.length  > maxSynonyms) randomize(stack);
    index = 0;
    current = token;
    todo = maxSynonyms;
    return token;
}

Returns the next token in the stream, or null at EOS.