Class: Kotoshu::Language::Tokenizer::ScriptTokenizer

Inherits:
Base
  • Object
show all
Defined in:
lib/kotoshu/language/tokenizer/script_tokenizer.rb,
sig/kotoshu.rbs

Overview

Regex-parameterized tokenizer for scripts without a dedicated class (plan 107 basic tier).

Instance Method Summary collapse

Methods inherited from Base

#normalize, #skip_token?, #tokenize_with_positions, #word_char?

Constructor Details

#initialize(word_regex:) ⇒ ScriptTokenizer

Returns a new instance of ScriptTokenizer.

Parameters:

  • word_regex (Regexp) —

    characters forming a word, as a single-character regex (the plan-91 set)

  • word_regex: (Regexp) 


21
22
23
24
 
# File 'lib/kotoshu/language/tokenizer/script_tokenizer.rb', line 21

def initialize(word_regex:)
  super()
  @word_regex = word_regex
end

Instance Method Details

#spellcheck_word_regex ⇒ Regexp

Spell-check word characters (plan 91): the script set this tokenizer was built with.

Returns:

  • (Regexp) —

    Regex matching a single word character



47
48
49
 
# File 'lib/kotoshu/language/tokenizer/script_tokenizer.rb', line 47

def spellcheck_word_regex
  @word_regex
end

#tokenize(text) ⇒ Array<String>

Tokenize text into words by scanning runs of word characters.

Parameters:

  • text (String) —

    Text to tokenize

Returns:

  • (Array<String>) —

    Array of tokens



30
31
32
33
34
 
# File 'lib/kotoshu/language/tokenizer/script_tokenizer.rb', line 30

def tokenize(text)
  return [] if text.nil? || text.empty?

  text.scan(/#{@word_regex}+/).reject { |token| skip_token?(token) }
end

#word_boundary_regex ⇒ Regexp

Get word boundary regex.

Returns:

  • (Regexp) —

    Word boundary regex



39
40
41
 
# File 'lib/kotoshu/language/tokenizer/script_tokenizer.rb', line 39

def word_boundary_regex
  @word_regex
end