Class: SClust::Util::TokenizerFilter

Inherits:

Filter

Object
Filter
SClust::Util::TokenizerFilter

show all

Defined in:: lib/sclust/util/filters.rb

Direct Known Subclasses

DocumentTokenizer

Instance Method Summary collapse

#filter(document) ⇒ Object

Methods inherited from Filter

#after, #apply, #before, #initialize

Constructor Details

This class inherits a constructor from SClust::Util::Filter

Instance Method Details

#filter(document) ⇒ `Object`



163
164
165

# File 'lib/sclust/util/filters.rb', line 163

def filter(document)
    document.split(/[\s,\.\t!\?\(\)\{\}\[\]\t\r\n";':]+/m)
end