Class: OpenTox::Validation

Inherits:
Object show all
Includes:
OldOpenTox
Defined in:
lib/validation.rb

Instance Attribute Summary

Attributes included from OldOpenTox

#metadata, #uri

Class Method Summary collapse

Instance Method Summary collapse

Methods included from OldOpenTox

#delete, #initialize, #load_metadata

Class Method Details

.create_bootstrapping_validation(params, waiting_task = nil) ⇒ OpenTox::Validation

creates a bootstrapping validation, waits until it finishes, may take some time

Parameters:

  • params (Hash) —

    (required:algorithm_uri,dataset_uri,prediction_feature, optional:algorithm_params,random_seed(1))

  • waiting_task (OpenTox::Task, optional) (defaults to: nil) —

    (can be a OpenTox::Subtask as well), progress is updated accordingly

Returns:


73
74
75
76
77
 
# File 'lib/validation.rb', line 73

def self.create_bootstrapping_validation( params, waiting_task=nil )
  uri = OpenTox::RestClientWrapper.post( File.join($validation[:uri],"bootstrapping"),
    params,{:content_type => "text/uri-list"},waiting_task )
  Validation.new(wait_for_task(uri))
end

.create_training_test_split(params, waiting_task = nil) ⇒ OpenTox::Validation

creates a training test split validation, waits until it finishes, may take some time

Parameters:

  • params (Hash) —

    (required:algorithm_uri,dataset_uri,prediction_feature, optional:algorithm_params,split_ratio(0.67),random_seed(1))

  • waiting_task (OpenTox::Task, optional) (defaults to: nil) —

    (can be a OpenTox::Subtask as well), progress is updated accordingly

Returns:


53
54
55
56
57
 
# File 'lib/validation.rb', line 53

def self.create_training_test_split( params, waiting_task=nil )
  uri = OpenTox::RestClientWrapper.post( File.join($validation[:uri],"training_test_split"),
    params,{:content_type => "text/uri-list"},waiting_task )
  Validation.new(wait_for_task(uri))
end

.create_training_test_validation(params, waiting_task = nil) ⇒ OpenTox::Validation

creates a training test validation, waits until it finishes, may take some time

Parameters:

  • params (Hash) —

    (required:algorithm_uri,training_dataset_uri,prediction_feature,test_dataset_uri,optional:algorithm_params)

  • waiting_task (OpenTox::Task, optional) (defaults to: nil) —

    (can be a OpenTox::Subtask as well), progress is updated accordingly

Returns:


63
64
65
66
67
 
# File 'lib/validation.rb', line 63

def self.create_training_test_validation( params, waiting_task=nil )
  uri = OpenTox::RestClientWrapper.post( File.join($validation[:uri],"training_test_validation"),
    params,{:content_type => "text/uri-list"},waiting_task )
  Validation.new(wait_for_task(uri))
end

.find(uri) ⇒ OpenTox::Validation

find validation, raises error if not found

Parameters:

Returns:


30
31
32
33
34
 
# File 'lib/validation.rb', line 30

def self.find( uri )
  val = Validation.new(uri)
  val.load_metadata
  val
end

.from_cv_statistics(crossvalidation_uri) ⇒ OpenTox::Validation

creates a validation object from crossvaldiation statistics, raise error if not found (as crossvaldiation statistics are returned as an average valdidation over all folds)

Parameters:

  • crossvalidation_uri (String) —

    crossvalidation uri

Returns:


92
93
94
 
# File 'lib/validation.rb', line 92

def self.from_cv_statistics( crossvalidation_uri )
  find( File.join(crossvalidation_uri, 'statistics') )
end

.list(params = {}) ⇒ Array

returns a filtered list of validation uris

Parameters:

  • params (Hash, optional) (defaults to: {}) —

    validation-params to filter the uris (could be model, training_dataset, ..)

Returns:


39
40
41
42
43
44
45
46
47
 
# File 'lib/validation.rb', line 39

def self.list( params={} )
  filter_string = ""
  params.each do |k,v|
    filter_string += (filter_string.length==0 ? "?" : "&")
    v = v.to_s.gsub(/;/, "%3b") if v.to_s =~ /;/
    filter_string += k.to_s+"="+v.to_s
  end
  (OpenTox::RestClientWrapper.get($validation[:uri]+filter_string).split("\n"))
end

Instance Method Details

#confusion_matrix ⇒ Object

returns confusion matrix as array, predicted values are in rows -> 99 inactive compounds have been predicted as active

Examples:

[[nil,"active","moderate","inactive"],["active",1,3,99],["moderate",4,2,8],["inactive",3,8,6]]

100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
 
# File 'lib/validation.rb', line 100

def confusion_matrix
  raise "no classification statistics, probably a regression valdiation" unless @metadata[RDF::OT.classificationStatistics]
  matrix =  @metadata[RDF::OT.classificationStatistics][RDF::OT.confusionMatrix][RDF::OT.confusionMatrixCell]
  values = matrix.collect{|cell| cell[RDF::OT.confusionMatrixPredicted]}.uniq
  table = [[nil]+values]
  values.each do |c|
    table << [c]
    values.each do |r|
      matrix.each do |cell|
        if cell[RDF::OT.confusionMatrixPredicted]==c and cell[RDF::OT.confusionMatrixActual]==r
          table[-1] << cell[RDF::OT.confusionMatrixValue].to_f
          break
        end
      end
    end
  end
  table
end

#filter_metadata(min_confidence, min_num_predictions = nil, max_num_predictions = nil) ⇒ Object

filters the validation-predictions and returns validation-metadata with filtered statistics return [Hash] metadata

Parameters:

  • min_confidence (Float) —

    predictions with confidence < min_confidence are filtered out

  • min_num_predictions (Integer) (defaults to: nil) —

    optional, additional param to min_confidence, the top min_num_predictions are selected, even if confidence to low

  • max_num_predictions (Integer) (defaults to: nil) —

    returns the top max_num_predictions (with the highest confidence), not compatible to min_confidence


124
125
126
127
128
129
 
# File 'lib/validation.rb', line 124

def filter_metadata( min_confidence, min_num_predictions=nil, max_num_predictions=nil )
  conf = min_confidence ? "min_confidence=#{min_confidence}" : nil
  min = min_num_predictions ? "min_num_predictions=#{min_num_predictions}" : nil
  max = max_num_predictions ? "max_num_predictions=#{max_num_predictions}" : nil
  YAML.load(OpenTox::RestClientWrapper.get("#{@uri}?#{[conf,min,max].compact.join("&")}",nil,{:accept => "application/x-yaml"}))
end

#find_or_create_report(waiting_task = nil) ⇒ String

looks for report for this validation, creates a report if no report is found

Parameters:

  • waiting_task (OpenTox::Task, optional) (defaults to: nil) —

    (can be a OpenTox::Subtask as well), progress is updated accordingly

Returns:


82
83
84
85
86
 
# File 'lib/validation.rb', line 82

def find_or_create_report( waiting_task=nil )
  @report = ValidationReport.find_for_validation(@uri) unless @report
  @report = ValidationReport.create(@uri, waiting_task) unless @report
  @report.uri
end

#probabilities(confidence, prediction) ⇒ Hash

returns probability-distribution for a given prediction it takes all predictions into account that have a confidence value that is >= confidence and that have the same predicted value (minimum 12 predictions with the hightest confidence are selected (even if the confidence is lower than the given param)

Examples:

Example 1:
 validation.probabilities(0.3,"active")
 -> { :min_confidence=>0.32, :num_predictions=>20, :probs=>{"active"=>0.7, "moderate"=>0.25 "inactive"=>0.05 } }
there have been 20 "active" predictions with confidence >= 0.3, 70 percent of them beeing correct

Example 2:
 validation.probabilities(0.8,"active")
 -> { :min_confidence=>0.45, :num_predictions=>12, :probs=>{"active"=>0.9, "moderate"=>0.1 "inactive"=>0 } }
the given confidence value was to high (i.e. <12 predictions with confidence value >= 0.8)
the top 12 "active" predictions have a min_confidence of 0.45, 90 percent of them beeing correct

Parameters:

  • confidence (Float) —

    confidence value (between 0 and 1)

  • prediction (String) —

    predicted value

Returns:

  • (Hash) —

    see example


150
151
152
153
 
# File 'lib/validation.rb', line 150

def probabilities( confidence, prediction )
  YAML.load(OpenTox::RestClientWrapper.get(@uri+"/probabilities?prediction="+prediction.to_s+"&confidence="+confidence.to_s,nil,
    {:accept => "application/x-yaml"}))
end