Module: ClassifierReborn::TokenFilter::Symbol

Defined in:
lib/classifier-reborn/extensions/token_filter/symbol.rb

Overview

This filter removes symbol-only terms, from given tokens.

Class Method Summary collapse

Class Method Details

.call(tokens) ⇒ Object 



13
14
15
16
17
 
# File 'lib/classifier-reborn/extensions/token_filter/symbol.rb', line 13

def call(tokens)
  tokens.reject do |token|
    /[^\s\p{WORD}]/ === token
  end
end