Class: Ferret::Analysis::AsciiStandardAnalyzer

Inherits:

Object

Object
Ferret::Analysis::AsciiStandardAnalyzer

show all

Defined in:: ext/r_analysis.c

Overview

Summary

The AsciiStandardAnalyzer is the most advanced of the available ASCII-analyzers. If it were implemented in Ruby it would look like this;

class AsciiStandardAnalyzer
  def initialize(stop_words = FULL_ENGLISH_STOP_WORDS, lower = true)
    @lower = lower
    @stop_words = stop_words
  end

  def token_stream(field, str)
    ts = AsciiStandardTokenizer.new(str)
    ts = AsciiLowerCaseFilter.new(ts) if @lower
    ts = StopFilter.new(ts, @stop_words)
    ts = HyphenFilter.new(ts)
  end
end

As you can see it makes use of the AsciiStandardTokenizer and you can also add your own list of stop-words if you wish. Note that this tokenizer won’t recognize non-ASCII characters so you should use the StandardAnalyzer is you want to analyze multi-byte data like “UTF-8”.