Class: Ferret::Analysis::StandardTokenizer

Inherits:

Object

Object
Ferret::Analysis::StandardTokenizer

show all

Defined in:: ext/r_analysis.c

Overview

The standard tokenizer is an advanced tokenizer which tokenizes most words correctly as well as tokenizing things like email addresses, web addresses, phone numbers, etc.

Example

"Dave's résumé, at http://www.davebalmain.com/ 1234"
  => ["Dave's", "résumé", "at", "http://www.davebalmain.com", "1234"]