Class: Classifier::LogisticRegression

Inherits:
Object
  • Object
show all
Includes:
Streaming, Mutex_m
Defined in:
lib/classifier/logistic_regression.rb

Overview

Logistic Regression (MaxEnt) classifier using Stochastic Gradient Descent. Often provides better accuracy than Naive Bayes while remaining fast and interpretable.

Example:

classifier = Classifier::LogisticRegression.new(:spam, :ham)
classifier.train(spam: ["Buy now!", "Free money!!!"])
classifier.train(ham: ["Meeting tomorrow", "Project update"])
classifier.classify("Claim your prize!") # => "Spam"
classifier.probabilities("Claim your prize!") # => {"Spam" => 0.92, "Ham" => 0.08}

Constant Summary collapse

DEFAULT_LEARNING_RATE = 
0.1
DEFAULT_REGULARIZATION = 
0.01
DEFAULT_MAX_ITERATIONS = 
100
DEFAULT_TOLERANCE = 
1e-4

Constants included from Streaming

Streaming::DEFAULT_BATCH_SIZE

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Streaming

#delete_checkpoint, #list_checkpoints, #save_checkpoint

Constructor Details

#initialize(*categories, learning_rate: DEFAULT_LEARNING_RATE, regularization: DEFAULT_REGULARIZATION, max_iterations: DEFAULT_MAX_ITERATIONS, tolerance: DEFAULT_TOLERANCE) ⇒ LogisticRegression

Creates a new Logistic Regression classifier with the specified categories.

classifier = Classifier::LogisticRegression.new(:spam, :ham)
classifier = Classifier::LogisticRegression.new('Positive', 'Negative', 'Neutral')
classifier = Classifier::LogisticRegression.new(['Positive', 'Negative', 'Neutral'])

Options:

  • learning_rate: Step size for gradient descent (default: 0.1)

  • regularization: L2 regularization strength (default: 0.01)

  • max_iterations: Maximum training iterations (default: 100)

  • tolerance: Convergence threshold (default: 1e-4)



59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
 
# File 'lib/classifier/logistic_regression.rb', line 59

def initialize(*categories, learning_rate: DEFAULT_LEARNING_RATE,
               regularization: DEFAULT_REGULARIZATION,
               max_iterations: DEFAULT_MAX_ITERATIONS,
               tolerance: DEFAULT_TOLERANCE)
  super()
  categories = categories.flatten
  @categories = categories.map { |c| c.to_s.prepare_category_name }
  @weights = @categories.to_h { |c| [c, {}] }
  @bias = @categories.to_h { |c| [c, 0.0] }
  @vocabulary = {}
  @training_data = []
  @learning_rate = learning_rate
  @regularization = regularization
  @max_iterations = max_iterations
  @tolerance = tolerance
  @fitted = false
  @dirty = false
  @storage = nil
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method

#method_missing(name, *args) ⇒ Object

Provides training methods for the categories.

classifier.train_spam "Buy now!"

Raises:

  • (StandardError) 


210
211
212
213
214
215
216
217
218
 
# File 'lib/classifier/logistic_regression.rb', line 210

def method_missing(name, *args)
  category_match = name.to_s.match(/train_(\w+)/)
  return super unless category_match

  category = category_match[1].to_s.prepare_category_name
  raise StandardError, "No such category: #{category}" unless @categories.include?(category)

  args.each { |text| train(category, text) }
end

Instance Attribute Details

#storage ⇒ Object

Returns the value of attribute storage.



38
39
40
 
# File 'lib/classifier/logistic_regression.rb', line 38

def storage
  @storage
end

Class Method Details

.from_json(json) ⇒ Object

Loads a classifier from a JSON string or Hash.

Raises:

  • (ArgumentError) 


256
257
258
259
260
261
262
263
264
 
# File 'lib/classifier/logistic_regression.rb', line 256

def self.from_json(json)
  data = json.is_a?(String) ? JSON.parse(json) : json
  raise ArgumentError, "Invalid classifier type: #{data['type']}" unless data['type'] == 'logistic_regression'

  categories = data['categories'].map(&:to_sym)
  instance = allocate
  instance.send(:restore_state, data, categories)
  instance
end

.load(storage:) ⇒ Object

Loads a classifier from the configured storage.

Raises:



288
289
290
291
292
293
294
295
 
# File 'lib/classifier/logistic_regression.rb', line 288

def self.load(storage:)
  data = storage.read
  raise StorageError, 'No saved state found' unless data

  instance = from_json(data)
  instance.storage = storage
  instance
end

.load_checkpoint(storage:, checkpoint_id:) ⇒ Object

Loads a classifier from a checkpoint.

Raises:

  • (ArgumentError) 


357
358
359
360
361
362
363
364
365
366
367
368
369
 
# File 'lib/classifier/logistic_regression.rb', line 357

def self.load_checkpoint(storage:, checkpoint_id:)
  raise ArgumentError, 'Storage must be File storage for checkpoints' unless storage.is_a?(Storage::File)

  dir = File.dirname(storage.path)
  base = File.basename(storage.path, '.*')
  ext = File.extname(storage.path)
  checkpoint_path = File.join(dir, "#{base}_checkpoint_#{checkpoint_id}#{ext}")

  checkpoint_storage = Storage::File.new(path: checkpoint_path)
  instance = load(storage: checkpoint_storage)
  instance.storage = storage
  instance
end

.load_from_file(path) ⇒ Object

Loads a classifier from a file.



300
301
302
 
# File 'lib/classifier/logistic_regression.rb', line 300

def self.load_from_file(path)
  from_json(File.read(path))
end

Instance Method Details

#add_category(category) ⇒ Object

Adds a new category to the classifier. Allows dynamic category creation for CLI and incremental training.



181
182
183
184
185
186
187
188
189
190
191
192
 
# File 'lib/classifier/logistic_regression.rb', line 181

def add_category(category)
  cat = category.to_s.prepare_category_name
  synchronize do
    return if @categories.include?(cat)

    @categories << cat
    @weights[cat] = {}
    @bias[cat] = 0.0
    @fitted = false
    @dirty = true
  end
end

#as_json(_options = nil) ⇒ Object

Returns a hash representation of the classifier state. Does NOT auto-fit; saves current state including unfitted models.



229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
 
# File 'lib/classifier/logistic_regression.rb', line 229

def as_json(_options = nil)
  {
    version: 1,
    type: 'logistic_regression',
    categories: @categories.map(&:to_s),
    weights: @weights.transform_keys(&:to_s).transform_values { |v| v.transform_keys(&:to_s) },
    bias: @bias.transform_keys(&:to_s),
    vocabulary: @vocabulary.keys.map(&:to_s),
    training_data: @training_data.map { |d| { category: d[:category].to_s, features: d[:features].transform_keys(&:to_s) } },
    learning_rate: @learning_rate,
    regularization: @regularization,
    max_iterations: @max_iterations,
    tolerance: @tolerance,
    fitted: @fitted
  }
end

#categories ⇒ Object

Returns the list of categories.



173
174
175
 
# File 'lib/classifier/logistic_regression.rb', line 173

def categories
  synchronize { @categories.map(&:to_s) }
end

#classifications(text) ⇒ Object

Returns log-odds scores for each category (before softmax). Raises NotFittedError if model has not been fitted.

Raises:



143
144
145
146
147
148
149
150
 
# File 'lib/classifier/logistic_regression.rb', line 143

def classifications(text)
  raise NotFittedError, 'Model not fitted. Call fit() after training.' unless @fitted

  features = text.word_hash
  synchronize do
    compute_scores(features).transform_keys(&:to_s)
  end
end

#classify(text) ⇒ Object

Returns the best matching category for the provided text.

classifier.classify("Buy now!") # => "Spam"

Raises:

  • (StandardError) 


114
115
116
117
118
119
120
 
# File 'lib/classifier/logistic_regression.rb', line 114

def classify(text)
  probs = probabilities(text)
  best = probs.max_by { |_, v| v }
  raise StandardError, 'No classifications available' unless best

  best.first
end

#dirty? ⇒ Boolean

Returns true if there are unsaved changes.

Returns:

  • (Boolean) 


204
205
206
 
# File 'lib/classifier/logistic_regression.rb', line 204

def dirty?
  @dirty
end

#fit ⇒ Object

Fits the model to all accumulated training data. Called automatically during classify/probabilities if not already fitted.



97
98
99
100
101
102
103
104
105
106
107
 
# File 'lib/classifier/logistic_regression.rb', line 97

def fit
  synchronize do
    return self if @training_data.empty?
    raise ArgumentError, 'At least two categories required for fitting' if @categories.size < 2

    optimize_weights
    @fitted = true
    @dirty = false
  end
  self
end

#fitted? ⇒ Boolean

Returns true if the model has been fitted.

Returns:

  • (Boolean) 


197
198
199
 
# File 'lib/classifier/logistic_regression.rb', line 197

def fitted?
  @fitted
end

#marshal_dump ⇒ Object

Custom marshal serialization to exclude mutex state.



336
337
338
339
340
 
# File 'lib/classifier/logistic_regression.rb', line 336

def marshal_dump
  fit unless @fitted
  [@categories, @weights, @bias, @vocabulary, @learning_rate, @regularization,
   @max_iterations, @tolerance, @fitted]
end

#marshal_load(data) ⇒ Object

Custom marshal deserialization to recreate mutex.



345
346
347
348
349
350
351
352
 
# File 'lib/classifier/logistic_regression.rb', line 345

def marshal_load(data)
  mu_initialize
  @categories, @weights, @bias, @vocabulary, @learning_rate, @regularization,
    @max_iterations, @tolerance, @fitted = data
  @training_data = []
  @dirty = false
  @storage = nil
end

#probabilities(text) ⇒ Object

Returns probability distribution across all categories. Probabilities are well-calibrated (unlike Naive Bayes). Raises NotFittedError if model has not been fitted.

classifier.probabilities("Buy now!")
# => {"Spam" => 0.92, "Ham" => 0.08}

Raises:



130
131
132
133
134
135
136
137
 
# File 'lib/classifier/logistic_regression.rb', line 130

def probabilities(text)
  raise NotFittedError, 'Model not fitted. Call fit() after training.' unless @fitted

  features = text.word_hash
  synchronize do
    softmax(compute_scores(features))
  end
end

#reload ⇒ Object

Reloads the classifier from storage, raising if there are unsaved changes.

Raises:

  • (ArgumentError) 


307
308
309
310
311
312
313
314
315
316
317
 
# File 'lib/classifier/logistic_regression.rb', line 307

def reload
  raise ArgumentError, 'No storage configured' unless storage
  raise UnsavedChangesError, 'Unsaved changes would be lost. Call save first or use reload!' if @dirty

  data = storage.read
  raise StorageError, 'No saved state found' unless data

  restore_from_json(data)
  @dirty = false
  self
end

#reload! ⇒ Object

Force reloads the classifier from storage, discarding any unsaved changes.

Raises:

  • (ArgumentError) 


322
323
324
325
326
327
328
329
330
331
 
# File 'lib/classifier/logistic_regression.rb', line 322

def reload!
  raise ArgumentError, 'No storage configured' unless storage

  data = storage.read
  raise StorageError, 'No saved state found' unless data

  restore_from_json(data)
  @dirty = false
  self
end

#respond_to_missing?(name, include_private = false) ⇒ Boolean

Returns:

  • (Boolean) 


221
222
223
 
# File 'lib/classifier/logistic_regression.rb', line 221

def respond_to_missing?(name, include_private = false)
  !!(name.to_s =~ /train_(\w+)/) || super
end

#save ⇒ Object

Saves the classifier to the configured storage.

Raises:

  • (ArgumentError) 


269
270
271
272
273
274
 
# File 'lib/classifier/logistic_regression.rb', line 269

def save
  raise ArgumentError, 'No storage configured' unless storage

  storage.write(to_json)
  @dirty = false
end

#save_to_file(path) ⇒ Object

Saves the classifier state to a file.



279
280
281
282
283
 
# File 'lib/classifier/logistic_regression.rb', line 279

def save_to_file(path)
  result = File.write(path, to_json)
  @dirty = false
  result
end

#to_json(_options = nil) ⇒ Object

Serializes the classifier state to a JSON string.



249
250
251
 
# File 'lib/classifier/logistic_regression.rb', line 249

def to_json(_options = nil)
  JSON.generate(as_json)
end

#train(category = nil, text = nil, **categories) ⇒ Object

Trains the classifier with text for a category.

classifier.train(spam: "Buy now!", ham: ["Hello", "Meeting tomorrow"])
classifier.train(:spam, "legacy positional API")



85
86
87
88
89
90
91
 
# File 'lib/classifier/logistic_regression.rb', line 85

def train(category = nil, text = nil, **categories)
  return train_single(category, text) if category && text

  categories.each do |cat, texts|
    (texts.is_a?(Array) ? texts : [texts]).each { |t| train_single(cat, t) }
  end
end

#train_batch(category = nil, documents = nil, batch_size: Streaming::DEFAULT_BATCH_SIZE, **categories, &block) ⇒ Object

Trains the classifier with an array of documents in batches. Note: The model is NOT automatically fitted after batch training. Call #fit to train the model after adding all data.

Examples:

Positional style

classifier.train_batch(:spam, documents, batch_size: 100)
classifier.fit

Keyword style

classifier.train_batch(spam: documents, ham: other_docs)
classifier.fit



424
425
426
427
428
429
430
431
432
 
# File 'lib/classifier/logistic_regression.rb', line 424

def train_batch(category = nil, documents = nil, batch_size: Streaming::DEFAULT_BATCH_SIZE, **categories, &block)
  if category && documents
    train_batch_for_category(category, documents, batch_size: batch_size, &block)
  else
    categories.each do |cat, docs|
      train_batch_for_category(cat, Array(docs), batch_size: batch_size, &block)
    end
  end
end

#train_from_stream(category, io, batch_size: Streaming::DEFAULT_BATCH_SIZE) ⇒ Object

Trains the classifier from an IO stream. Each line in the stream is treated as a separate document. Note: The model is NOT automatically fitted after streaming. Call #fit to train the model after adding all data.

Examples:

Train from a file

classifier.train_from_stream(:spam, File.open('spam_corpus.txt'))
classifier.fit  # Required to train the model

With progress tracking

classifier.train_from_stream(:spam, io, batch_size: 500) do |progress|
  puts "#{progress.completed} documents processed"
end
classifier.fit

Raises:

  • (StandardError) 


387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
 
# File 'lib/classifier/logistic_regression.rb', line 387

def train_from_stream(category, io, batch_size: Streaming::DEFAULT_BATCH_SIZE)
  category = category.to_s.prepare_category_name
  raise StandardError, "No such category: #{category}" unless @categories.include?(category)

  reader = Streaming::LineReader.new(io, batch_size: batch_size)
  total = reader.estimate_line_count
  progress = Streaming::Progress.new(total: total)

  reader.each_batch do |batch|
    synchronize do
      batch.each do |text|
        features = text.word_hash
        features.each_key { |word| @vocabulary[word] = true }
        @training_data << { category: category, features: features }
      end
      @fitted = false
      @dirty = true
    end
    progress.completed += batch.size
    progress.current_batch += 1
    yield progress if block_given?
  end
end

#weights(category, limit: nil) ⇒ Object

Returns feature weights for a category, sorted by importance. Positive weights indicate the feature supports the category.

classifier.weights(:spam)
# => {:free => 2.3, :buy => 1.8, :money => 1.5, ...}

Raises:

  • (StandardError) 


159
160
161
162
163
164
165
166
167
168
 
# File 'lib/classifier/logistic_regression.rb', line 159

def weights(category, limit: nil)
  fit unless @fitted

  cat = category.to_s.prepare_category_name
  raise StandardError, "No such category: #{cat}" unless @weights.key?(cat)

  sorted = @weights[cat].sort_by { |_, v| -v.abs }
  sorted = sorted.first(limit) if limit
  sorted.to_h
end