Class: SVMKit::ModelSelection::CrossValidation

Inherits:
Object
  • Object
show all
Defined in:
lib/svmkit/model_selection/cross_validation.rb

Overview

CrossValidation is a class that evaluates a given classifier with cross-validation method.

Examples:

svc = SVMKit::LinearModel::SVC.new
kf = SVMKit::ModelSelection::StratifiedKFold.new(n_splits: 5)
cv = SVMKit::ModelSelection::CrossValidation.new(estimator: svc, splitter: kf)
report = cv.perform(samples, lables)
mean_test_score = report[:test_score].inject(:+) / kf.n_splits

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(estimator: nil, splitter: nil, evaluator: nil, return_train_score: false) ⇒ CrossValidation

Create a new evaluator with cross-validation method.

Parameters:

  • estimator (Classifier) (defaults to: nil) —

    The classifier of which performance is evaluated.

  • splitter (Splitter) (defaults to: nil) —

    The splitter that divides dataset to training and testing dataset.

  • evaluator (Evaluator) (defaults to: nil) —

    The evaluator that calculates score of estimator results.

  • return_train_score (Boolean) (defaults to: false) —

    The flag indicating whether to calculate the score of training dataset.



40
41
42
43
44
45
 
# File 'lib/svmkit/model_selection/cross_validation.rb', line 40

def initialize(estimator: nil, splitter: nil, evaluator: nil, return_train_score: false)
  @estimator = estimator
  @splitter = splitter
  @evaluator = evaluator
  @return_train_score = return_train_score
end

Instance Attribute Details

#estimator ⇒ Classifier (readonly)

Return the classifier of which performance is evaluated.

Returns:

  • (Classifier) 


20
21
22
 
# File 'lib/svmkit/model_selection/cross_validation.rb', line 20

def estimator
  @estimator
end

#evaluator ⇒ Evaluator (readonly)

Return the evaluator that calculates score.

Returns:

  • (Evaluator) 


28
29
30
 
# File 'lib/svmkit/model_selection/cross_validation.rb', line 28

def evaluator
  @evaluator
end

#return_train_score ⇒ Boolean (readonly)

Return the flag indicating whether to caculate the score of training dataset.

Returns:

  • (Boolean) 


32
33
34
 
# File 'lib/svmkit/model_selection/cross_validation.rb', line 32

def return_train_score
  @return_train_score
end

#splitter ⇒ Splitter (readonly)

Return the splitter that divides dataset.

Returns:

  • (Splitter) 


24
25
26
 
# File 'lib/svmkit/model_selection/cross_validation.rb', line 24

def splitter
  @splitter
end

Instance Method Details

#perform(x, y) ⇒ Hash

Perform the evalution of given classifier with cross-validation method.

Parameters:

  • x (Numo::DFloat) —

    (shape: [n_samples, n_features]) The dataset to be used to evaluate the classifier.

  • y (Numo::Int32) —

    (shape: [n_samples]) The labels to be used to evaluate the classifier.

Returns:

  • (Hash) —

    The report summarizing the results of cross-validation.

    • :fit_time (Array<Float>) The calculation times of fitting the estimator for each split.

    • :test_score (Array<Float>) The scores of testing dataset for each split.

    • :train_score (Array<Float>) The scores of training dataset for each split. This option is nil if the return_train_score is false.



58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
 
# File 'lib/svmkit/model_selection/cross_validation.rb', line 58

def perform(x, y)
  # Initialize the report of cross validation.
  report = { test_score: [], train_score: nil, fit_time: [] }
  report[:train_score] = [] if @return_train_score
  # Evaluate the estimator on each split.
  @splitter.split(x, y).each do |train_ids, test_ids|
    # Split dataset into training and testing dataset.
    feature_ids = !kernel_machine? || train_ids
    train_x = x[train_ids, feature_ids]
    train_y = y[train_ids]
    test_x = x[test_ids, feature_ids]
    test_y = y[test_ids]
    # Fit the estimator.
    start_time = Time.now.to_i
    @estimator.fit(train_x, train_y)
    # Calculate scores and prepare the report.
    report[:fit_time].push(Time.now.to_i - start_time)
    if @evaluator.nil?
      report[:test_score].push(@estimator.score(test_x, test_y))
      report[:train_score].push(@estimator.score(train_x, train_y)) if @return_train_score
    else
      report[:test_score].push(@evaluator.score(test_y, @estimator.predict(test_x)))
      report[:train_score].push(@estimator.score(train_x, @estimator.predict(train_x))) if @return_train_score
    end
  end
  report
end