Class: Bishop::Tokenizer
- Inherits:
-
Object
- Object
- Bishop::Tokenizer
- Defined in:
- lib/bishop.rb
Instance Method Summary collapse
Instance Method Details
#tokenize(item) ⇒ Object
44 45 46 47 48 |
# File 'lib/bishop.rb', line 44 def tokenize( item ) item.split( /\s+/ ).map do |i| token = i.split( /\-/ ).map { |token| token.downcase.gsub( /\W/, "" ) }.join( "-" ) end.reject { |t| t == "" || t == "-" } end |