Module: Scientist::Experiment

Included in:
Default
Defined in:
lib/scientist/experiment.rb

Overview

This mixin provides shared behavior for experiments. Includers must implement ‘enabled?` and `publish(result)`.

Override Scientist::Experiment.new to set your own class which includes and implements Scientist::Experiment’s interface.

Defined Under Namespace

Modules: RaiseOnMismatch Classes: MismatchError

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#raise_on_mismatches ⇒ Object

Whether to raise when the control and candidate mismatch. If this is nil, raise_on_mismatches class attribute is used instead.



10
11
12
 
# File 'lib/scientist/experiment.rb', line 10

def raise_on_mismatches
  @raise_on_mismatches
end

Class Method Details

.included(base) ⇒ Object 



12
13
14
15
 
# File 'lib/scientist/experiment.rb', line 12

def self.included(base)
  self.set_default(base) if base.instance_of?(Class)
  base.extend RaiseOnMismatch
end

.new(name) ⇒ Object

Instantiate a new experiment (using the class given to the .set_default method).



18
19
20
 
# File 'lib/scientist/experiment.rb', line 18

def self.new(name)
  (@experiment_klass || Scientist::Default).new(name)
end

.set_default(klass) ⇒ Object

Configure Scientist to use the given class for all future experiments (must implement the Scientist::Experiment interface).

Called automatically when new experiments are defined.



26
27
28
 
# File 'lib/scientist/experiment.rb', line 26

def self.set_default(klass)
  @experiment_klass = klass
end

Instance Method Details

#after_run(&block) ⇒ Object

Define a block of code to run after an experiment completes, if the experiment is enabled.

The block takes one argument, the Scientist::Result containing experiment results.

Returns the configured block.



96
97
98
 
# File 'lib/scientist/experiment.rb', line 96

def after_run(&block)
  @_scientist_after_run = block
end

#before_run(&block) ⇒ Object

Define a block of code to run before an experiment begins, if the experiment is enabled.

The block takes no arguments.

Returns the configured block.



86
87
88
 
# File 'lib/scientist/experiment.rb', line 86

def before_run(&block)
  @_scientist_before_run = block
end

#behaviors ⇒ Object

A Hash of behavior blocks, keyed by String name. Register behavior blocks with the ‘try` and `use` methods.



102
103
104
 
# File 'lib/scientist/experiment.rb', line 102

def behaviors
  @_scientist_behaviors ||= {}
end

#clean(&block) ⇒ Object

A block to clean an observed value for publishing or storing.

The block takes one argument, the observed value which will be cleaned.

Returns the configured block.



111
112
113
 
# File 'lib/scientist/experiment.rb', line 111

def clean(&block)
  @_scientist_cleaner = block
end

#clean_value(value) ⇒ Object

Internal: Clean a value with the configured clean block, or return the value if no clean block is configured.

Rescues and reports exceptions in the clean block if they occur.



126
127
128
129
130
131
132
133
134
135
 
# File 'lib/scientist/experiment.rb', line 126

def clean_value(value)
  if @_scientist_cleaner
    @_scientist_cleaner.call value
  else
    value
  end
rescue StandardError => ex
  raised :clean, ex
  value
end

#cleaner ⇒ Object

Accessor for the clean block, if one is available.

Returns the configured block, or nil.



118
119
120
 
# File 'lib/scientist/experiment.rb', line 118

def cleaner
  @_scientist_cleaner
end

#compare(*args, &block) ⇒ Object

A block which compares two experimental values.

The block must take two arguments, the control value and a candidate value, and return true or false.

Returns the block.



143
144
145
 
# File 'lib/scientist/experiment.rb', line 143

def compare(*args, &block)
  @_scientist_comparator = block
end

#compare_errors(*args, &block) ⇒ Object

A block which compares two experimental errors.

The block must take two arguments, the control Error and a candidate Error, and return true or false.

Returns the block.



153
154
155
 
# File 'lib/scientist/experiment.rb', line 153

def compare_errors(*args, &block)
  @_scientist_error_comparator = block
end

#context(context = nil) ⇒ Object

A Symbol-keyed Hash of extra experiment data.



158
159
160
161
162
 
# File 'lib/scientist/experiment.rb', line 158

def context(context = nil)
  @_scientist_context ||= {}
  @_scientist_context.merge!(context) unless context.nil?
  @_scientist_context
end

#fabricate_durations_for_testing_purposes(fabricated_durations = {}) ⇒ Object

Provide predefined durations to use instead of actual timing data. This is here solely as a convenience for developers of libraries that extend Scientist.



318
319
320
 
# File 'lib/scientist/experiment.rb', line 318

def fabricate_durations_for_testing_purposes(fabricated_durations = {})
  @_scientist_fabricated_durations = fabricated_durations
end

#generate_result(name) ⇒ Object

Internal: Generate the observations and create the result from those and the control.



323
324
325
326
327
328
329
330
331
332
333
334
 
# File 'lib/scientist/experiment.rb', line 323

def generate_result(name)
  observations = []

  behaviors.keys.shuffle.each do |key|
    block = behaviors[key]
    fabricated_duration = @_scientist_fabricated_durations && @_scientist_fabricated_durations[key]
    observations << Scientist::Observation.new(key, self, fabricated_duration: fabricated_duration, &block)
  end

  control = observations.detect { |o| o.name == name }
  Scientist::Result.new(self, observations, control)
end

#ignore(&block) ⇒ Object

Configure this experiment to ignore an observation with the given block.

The block takes two arguments, the control observation and the candidate observation which didn’t match the control. If the block returns true, the mismatch is disregarded.

This can be called more than once with different blocks to use.



171
172
173
174
 
# File 'lib/scientist/experiment.rb', line 171

def ignore(&block)
  @_scientist_ignores ||= []
  @_scientist_ignores << block
end

#ignore_mismatched_observation?(control, candidate) ⇒ Boolean

Internal: ignore a mismatched observation?

Iterates through the configured ignore blocks and calls each of them with the given control and mismatched candidate observations.

Returns true or false.

Returns:

  • (Boolean) 


182
183
184
185
186
187
188
189
190
191
192
 
# File 'lib/scientist/experiment.rb', line 182

def ignore_mismatched_observation?(control, candidate)
  return false unless @_scientist_ignores
  @_scientist_ignores.any? do |ignore|
    begin
      ignore.call control.value, candidate.value
    rescue StandardError => ex
      raised :ignore, ex
      false
    end
  end
end

#name ⇒ Object

The String name of this experiment. Default is “experiment”. See Scientist::Default for an example of how to override this default.



196
197
198
 
# File 'lib/scientist/experiment.rb', line 196

def name
  "experiment"
end

#observations_are_equivalent?(a, b) ⇒ Boolean

Internal: compare two observations, using the configured compare and compare_errors lambdas if present.

Returns:

  • (Boolean) 


201
202
203
204
205
206
 
# File 'lib/scientist/experiment.rb', line 201

def observations_are_equivalent?(a, b)
  a.equivalent_to? b, @_scientist_comparator, @_scientist_error_comparator
rescue StandardError => ex
  raised :compare, ex
  false
end

#raise_on_mismatches? ⇒ Boolean

Whether or not to raise a mismatch error when a mismatch occurs.

Returns:

  • (Boolean) 


308
309
310
311
312
313
314
 
# File 'lib/scientist/experiment.rb', line 308

def raise_on_mismatches?
  if raise_on_mismatches.nil?
    self.class.raise_on_mismatches?
  else
    !!raise_on_mismatches
  end
end

#raise_with(exception) ⇒ Object 



208
209
210
 
# File 'lib/scientist/experiment.rb', line 208

def raise_with(exception)
  @_scientist_custom_mismatch_error = exception
end

#raised(operation, error) ⇒ Object

Called when an exception is raised while running an internal operation, like :publish. Override this method to track these exceptions. The default implementation re-raises the exception.



215
216
217
 
# File 'lib/scientist/experiment.rb', line 215

def raised(operation, error)
  raise error
end

#run(name = nil) ⇒ Object

Internal: Run all the behaviors for this experiment, observing each and publishing the results. Return the result of the named behavior, default “control”.



222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
 
# File 'lib/scientist/experiment.rb', line 222

def run(name = nil)
  behaviors.freeze
  context.freeze

  name = (name || "control").to_s
  block = behaviors[name]

  if block.nil?
    raise Scientist::BehaviorMissing.new(self, name)
  end

  unless should_experiment_run?
    return block.call
  end

  if @_scientist_before_run
    @_scientist_before_run.call
  end

  result = generate_result(name)

  if @_scientist_after_run
    @_scientist_after_run.call(result)
  end

  begin
    publish(result)
  rescue StandardError => ex
    raised :publish, ex
  end

  if raise_on_mismatches? && result.mismatched?
    if @_scientist_custom_mismatch_error
      raise @_scientist_custom_mismatch_error.new(self.name, result)
    else
      raise MismatchError.new(self.name, result)
    end
  end

  control = result.control
  raise control.exception if control.raised?
  control.value
end

#run_if(&block) ⇒ Object

Define a block that determines whether or not the experiment should run.



267
268
269
 
# File 'lib/scientist/experiment.rb', line 267

def run_if(&block)
  @_scientist_run_if_block = block
end

#run_if_block_allows? ⇒ Boolean

Internal: does a run_if block allow the experiment to run?

Rescues and reports exceptions in a run_if block if they occur.

Returns:

  • (Boolean) 


274
275
276
277
278
279
 
# File 'lib/scientist/experiment.rb', line 274

def run_if_block_allows?
  (@_scientist_run_if_block ? @_scientist_run_if_block.call : true)
rescue StandardError => ex
  raised :run_if, ex
  return false
end

#should_experiment_run? ⇒ Boolean

Internal: determine whether or not an experiment should run.

Rescues and reports exceptions in the enabled method if they occur.

Returns:

  • (Boolean) 


284
285
286
287
288
289
 
# File 'lib/scientist/experiment.rb', line 284

def should_experiment_run?
  behaviors.size > 1 && enabled? && run_if_block_allows?
rescue StandardError => ex
  raised :enabled, ex
  return false
end

#try(name = nil, &block) ⇒ Object

Register a named behavior for this experiment, default “candidate”.



292
293
294
295
296
297
298
299
300
 
# File 'lib/scientist/experiment.rb', line 292

def try(name = nil, &block)
  name = (name || "candidate").to_s

  if behaviors.include?(name)
    raise Scientist::BehaviorNotUnique.new(self, name)
  end

  behaviors[name] = block
end

#use(&block) ⇒ Object

Register the control behavior for this experiment.



303
304
305
 
# File 'lib/scientist/experiment.rb', line 303

def use(&block)
  try "control", &block
end