Class: Kotoshu::Algorithms::Lookup::Lookuper

Inherits:
Object
  • Object
show all
Includes:
CompoundChecks
Defined in:
lib/kotoshu/algorithms/lookup.rb

Overview

Main word correctness lookup class.

Typically, you would not use this directly.

Example:

dictionary = Kotoshu::Dictionary.load('en_US')
lookuper = dictionary.lookuper

lookuper.call('spylls')  # => false
lookuper.call('spells')  # => true

lookuper.good_forms('spells') do |form|
puts form
end
# AffixForm(spells = spells)
# AffixForm(spells = spell + Suffix(s: S×, on [[^sxzhy]]$))

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(aff, dic) ⇒ Lookuper

Returns a new instance of Lookuper.



60
61
62
63
 
# File 'lib/kotoshu/algorithms/lookup.rb', line 60

def initialize(aff, dic)
  @aff = aff
  @dic = dic
end

Instance Attribute Details

#aff ⇒ Hash (readonly)

Returns Aff data structure (from aff file).

Returns:

  • (Hash) —

    Aff data structure (from aff file)



55
56
57
 
# File 'lib/kotoshu/algorithms/lookup.rb', line 55

def aff
  @aff
end

#dic ⇒ Hash (readonly)

Returns Dic data structure (from dic file).

Returns:

  • (Hash) —

    Dic data structure (from dic file)



58
59
60
 
# File 'lib/kotoshu/algorithms/lookup.rb', line 58

def dic
  @dic
end

Instance Method Details

#break_word(text, depth = 0) {|Array<String>| ... } ⇒ Enumerator

Recursively produce all possible lists of word breaking by break patterns (like dashes).

Example: "pre-processed-meat" would produce:

["pre-processed-meat"]
["pre", "processed-meat"]
["pre", "processed", "meat"]
["pre-processed", "meat"]

This is necessary because dictionary might contain "pre-processed" as a separate entry.

Parameters:

  • text (String) —

    Text to break

  • depth (Integer) (defaults to: 0) —

    Current recursion depth

Yields:

  • (Array<String>) —

    Each possible breaking

Returns:

  • (Enumerator) —

    If no block given



124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
 
# File 'lib/kotoshu/algorithms/lookup.rb', line 124

def break_word(text, depth = 0)
  return enum_for(:break_word, text, depth) unless block_given?
  return if depth > 10

  # Return whole text as first option
  yield [text]

  break_patterns = @aff[:BREAK] || []
  break_patterns.each do |pattern|
    str = text.to_s
    pos = 0

    while (match_data = pattern[:matcher].match(str, pos))
      start = str[0...match_data.begin(1)]
      rest = str[match_data.end(1)..]

      break_word(rest, depth + 1) do |breaking|
        yield [start, *breaking]
      end

      pos = match_data.end(0)
      break if pos >= str.length
    end
  end
end

#call(word, capitalization: true, allow_nosuggest: true) ⇒ Boolean

The outermost word correctness check.

Basically, prepares word for check (converting/removing chars), and then checks whether any good word form can be produced with good_forms. If there is none, also tries to break word by break-points.

Parameters:

  • word (String) —

    Word to check

  • capitalization (Boolean) (defaults to: true) —

    If false, check only exact capitalization

  • allow_nosuggest (Boolean) (defaults to: true) —

    If false, don't consider NOSUGGEST words as correct

Returns:

  • (Boolean) —

    Whether word is correct



75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
 
# File 'lib/kotoshu/algorithms/lookup.rb', line 75

def call(word, capitalization: true, allow_nosuggest: true)
  # Check if word is correct
  is_correct = ->(w) do
    good_forms(w, capitalization: capitalization, allow_nosuggest: allow_nosuggest).any?
  end

  # If all entries matching the word have FORBIDDENWORD flag, word can't be correct
  if @aff[:FORBIDDENWORD] && @dic[:has_flag]&.call(word, @aff[:FORBIDDENWORD], for_all: true)
    return false
  end

  # Convert word with ICONV table
  word_to_check = @aff[:ICONV] ? @aff[:ICONV].call(word) : word

  # Remove ignored characters
  if @aff[:IGNORE]
    ignore_chars = @aff[:IGNORE]
    word_to_check = word_to_check.chars.reject { |c| ignore_chars.include?(c) }.join
  end

  # Numbers are always good
  return true if NUMBER_REGEXP.match?(word_to_check)

  # Try breaking word by break patterns
  break_word(word_to_check).each do |parts|
    if parts.all? { |part| part.empty? || is_correct.call(part) }
      return true
    end
  end

  false
end

#correct?(word, capitalization: true, allow_nosuggest: true, affix_forms: true, compound_forms: true) ⇒ Boolean Also known as: is_correct?

Check if the word is correct without yielding forms.

Convenience method for simple correctness checks.

Parameters:

  • word (String) —

    Word to check

  • capitalization (Boolean) (defaults to: true) —

    Check capitalization variants

  • allow_nosuggest (Boolean) (defaults to: true) —

    Include NOSUGGEST words

  • affix_forms (Boolean) (defaults to: true) —

    Check affix forms

  • compound_forms (Boolean) (defaults to: true) —

    Check compound forms

Returns:

  • (Boolean) —

    Whether word is correct



224
225
226
227
228
229
230
231
232
233
234
 
# File 'lib/kotoshu/algorithms/lookup.rb', line 224

def correct?(word,
             capitalization: true,
             allow_nosuggest: true,
             affix_forms: true,
             compound_forms: true)
  good_forms(word,
             capitalization: capitalization,
             allow_nosuggest: allow_nosuggest,
             affix_forms: affix_forms,
             compound_forms: compound_forms).any?
end

#good_forms(word, capitalization: true, allow_nosuggest: true, affix_forms: true, compound_forms: true) {|AffixForm, CompoundForm| ... } ⇒ Object

The main producer of correct word forms.

Produces all ways the proposed string might correspond to dictionary/ affixes. If there is at least one, the word is correctly spelled.

Example:

lookuper.good_forms('building') do |form|
puts form
end
# AffixForm(building = building)                              # noun
# AffixForm(building = build + Suffix(ing: G×, on [[^e]]$))   # verb

Parameters:

  • word (String) —

    Word to check

  • capitalization (Boolean) (defaults to: true) —

    If false, use only exact capitalization

  • allow_nosuggest (Boolean) (defaults to: true) —

    If false, exclude NOSUGGEST words

  • affix_forms (Boolean) (defaults to: true) —

    If false, only return compound forms

  • compound_forms (Boolean) (defaults to: true) —

    If false, only return affix forms

Yields:



168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
 
# File 'lib/kotoshu/algorithms/lookup.rb', line 168

def good_forms(word,
               capitalization: true,
               allow_nosuggest: true,
               affix_forms: true,
               compound_forms: true, &block)
  unless block
    return enum_for(:good_forms, word,
                    capitalization: capitalization,
                    allow_nosuggest: allow_nosuggest,
                    affix_forms: affix_forms,
                    compound_forms: compound_forms)
  end

  # Get capitalization variants
  if capitalization
    captype, variants = @aff[:casing].variants(word)
  else
    captype = @aff[:casing].guess(word)
    variants = [word]
  end

  # Check each variant
  variants.each do |variant|
    if affix_forms
      affix_forms_internal(variant, captype: captype, allow_nosuggest: allow_nosuggest) do |form|
        # Special German ß handling
        if @aff[:CHECKSHARPS] && @aff[:KEEPCASE]
          stem = form.in_dictionary ? form.in_dictionary[:stem] : form.stem
          if stem.include?('ß') &&
              captype == Capitalization::Type::ALL &&
              word.include?('ß') &&
              form.flags.include?(@aff[:KEEPCASE])
            next
          end
        end

        yield form
      end
    end

    if compound_forms
      compound_forms_internal(variant, captype: captype, allow_nosuggest: allow_nosuggest, &block)
    end
  end
end