Class: Minitest::Benchmark

Inherits:
Test show all
Defined in:
lib/minitest/benchmark.rb

Overview

Subclass Benchmark to create your own benchmark runs. Methods starting with “bench_” get executed on a per-class.

See Minitest::Assertions

Direct Known Subclasses

BenchSpec

Constant Summary

Constants inherited from Test

Test::PASSTHROUGH_EXCEPTIONS

Constants included from Assertions

Assertions::UNDEFINED

Instance Attribute Summary

Attributes inherited from Test

#time

Attributes inherited from Runnable

#assertions, #failures

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from Test

#capture_exceptions, #capture_io, #capture_subprocess_io, #error?, i_suck_and_my_tests_are_order_dependent!, #location, make_my_diffs_pretty!, #marshal_dump, #marshal_load, old_test_order, parallelize_me!, #passed?, #result_code, #run, #simple_capture_io, #simple_capture_subprocess_io, #skipped?, synchronize, test_order, #time_it, #to_s, #with_info_handler

Methods included from Guard

#jruby?, #maglev?, #mri?, #rubinius?, #windows?

Methods included from Test::LifecycleHooks

#after_setup, #after_teardown, #before_setup, #before_teardown, #setup, #teardown

Methods included from Assertions

#assert, #assert_empty, #assert_equal, #assert_in_delta, #assert_in_epsilon, #assert_includes, #assert_instance_of, #assert_kind_of, #assert_match, #assert_nil, #assert_operator, #assert_output, #assert_predicate, #assert_raises, #assert_respond_to, #assert_same, #assert_send, #assert_silent, #assert_throws, #capture_io, #capture_subprocess_io, #diff, diff, diff=, #exception_details, #flunk, #message, #mu_pp, #mu_pp_for_diff, #pass, #refute, #refute_empty, #refute_equal, #refute_in_delta, #refute_in_epsilon, #refute_includes, #refute_instance_of, #refute_kind_of, #refute_match, #refute_nil, #refute_operator, #refute_predicate, #refute_respond_to, #refute_same, #skip, #skipped?

Methods inherited from Runnable

#failure, inherited, #initialize, #marshal_dump, #marshal_load, methods_matching, #name, #name=, on_signal, #passed?, reset, #result_code, #run, runnables, #skipped?, with_info_handler

Constructor Details

This class inherits a constructor from Minitest::Runnable

Class Method Details

.bench_exp(min, max, base = 10) ⇒ Object

Returns a set of ranges stepped exponentially from min to max by powers of base. Eg:

bench_exp(2, 16, 2) # => [2, 4, 8, 16]



36
37
38
39
40
41
 
# File 'lib/minitest/benchmark.rb', line 36

def self.bench_exp min, max, base = 10
  min = (Math.log10(min) / Math.log10(base)).to_i
  max = (Math.log10(max) / Math.log10(base)).to_i

  (min..max).map { |m| base ** m }.to_a
end

.bench_linear(min, max, step = 10) ⇒ Object

Returns a set of ranges stepped linearly from min to max by step. Eg:

bench_linear(20, 40, 10) # => [20, 30, 40]



49
50
51
52
53
 
# File 'lib/minitest/benchmark.rb', line 49

def self.bench_linear min, max, step = 10
  (min..max).step(step).to_a
rescue LocalJumpError # 1.8.6
  r = []; (min..max).step(step) { |n| r << n }; r
end

.bench_range ⇒ Object

Specifies the ranges used for benchmarking for that class. Defaults to exponential growth from 1 to 10k by powers of 10. Override if you need different ranges for your benchmarks.

See also: ::bench_exp and ::bench_linear.



62
63
64
 
# File 'lib/minitest/benchmark.rb', line 62

def self.bench_range
  bench_exp 1, 10_000
end

.io ⇒ Object

:nodoc:



12
13
14
 
# File 'lib/minitest/benchmark.rb', line 12

def self.io # :nodoc:
  @io
end

.run(reporter, options = {}) ⇒ Object

:nodoc:



20
21
22
23
24
 
# File 'lib/minitest/benchmark.rb', line 20

def self.run reporter, options = {} # :nodoc:
  # NOTE: this is truly horrible... but I don't see a way around this ATM.
  @io = reporter.reporters.first.io
  super
end

.runnable_methods ⇒ Object

:nodoc:



26
27
28
 
# File 'lib/minitest/benchmark.rb', line 26

def self.runnable_methods # :nodoc:
  methods_matching(/^bench_/)
end

Instance Method Details

#assert_performance(validation, &work) ⇒ Object

Runs the given work, gathering the times of each run. Range and times are then passed to a given validation proc. Outputs the benchmark name and times in tab-separated format, making it easy to paste into a spreadsheet for graphing or further analysis.

Ranges are specified by ::bench_range.

Eg:

def bench_algorithm
  validation = proc { |x, y| ... }
  assert_performance validation do |n|
    @obj.algorithm(n)
  end
end



84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
 
# File 'lib/minitest/benchmark.rb', line 84

def assert_performance validation, &work
  range = self.class.bench_range

  io.print "#{self.name}"

  times = []

  range.each do |x|
    GC.start
    t0 = Time.now
    instance_exec(x, &work)
    t = Time.now - t0

    io.print "\t%9.6f" % t
    times << t
  end
  io.puts

  validation[range, times]
end

#assert_performance_constant(threshold = 0.99, &work) ⇒ Object

Runs the given work and asserts that the times gathered fit to match a constant rate (eg, linear slope == 0) within a given threshold. Note: because we’re testing for a slope of 0, R^2 is not a good determining factor for the fit, so the threshold is applied against the slope itself. As such, you probably want to tighten it from the default.

See www.graphpad.com/curvefit/goodness_of_fit.htm for more details.

Fit is calculated by #fit_linear.

Ranges are specified by ::bench_range.

Eg:

def bench_algorithm
  assert_performance_constant 0.9999 do |n|
    @obj.algorithm(n)
  end
end



128
129
130
131
132
133
134
135
136
 
# File 'lib/minitest/benchmark.rb', line 128

def assert_performance_constant threshold = 0.99, &work
  validation = proc do |range, times|
    a, b, rr = fit_linear range, times
    assert_in_delta 0, b, 1 - threshold
    [a, b, rr]
  end

  assert_performance validation, &work
end

#assert_performance_exponential(threshold = 0.99, &work) ⇒ Object

Runs the given work and asserts that the times gathered fit to match a exponential curve within a given error threshold.

Fit is calculated by #fit_exponential.

Ranges are specified by ::bench_range.

Eg:

def bench_algorithm
  assert_performance_exponential 0.9999 do |n|
    @obj.algorithm(n)
  end
end



154
155
156
 
# File 'lib/minitest/benchmark.rb', line 154

def assert_performance_exponential threshold = 0.99, &work
  assert_performance validation_for_fit(:exponential, threshold), &work
end

#assert_performance_linear(threshold = 0.99, &work) ⇒ Object

Runs the given work and asserts that the times gathered fit to match a straight line within a given error threshold.

Fit is calculated by #fit_linear.

Ranges are specified by ::bench_range.

Eg:

def bench_algorithm
  assert_performance_linear 0.9999 do |n|
    @obj.algorithm(n)
  end
end



194
195
196
 
# File 'lib/minitest/benchmark.rb', line 194

def assert_performance_linear threshold = 0.99, &work
  assert_performance validation_for_fit(:linear, threshold), &work
end

#assert_performance_logarithmic(threshold = 0.99, &work) ⇒ Object

Runs the given work and asserts that the times gathered fit to match a logarithmic curve within a given error threshold.

Fit is calculated by #fit_logarithmic.

Ranges are specified by ::bench_range.

Eg:

def bench_algorithm
  assert_performance_logarithmic 0.9999 do |n|
    @obj.algorithm(n)
  end
end



174
175
176
 
# File 'lib/minitest/benchmark.rb', line 174

def assert_performance_logarithmic threshold = 0.99, &work
  assert_performance validation_for_fit(:logarithmic, threshold), &work
end

#assert_performance_power(threshold = 0.99, &work) ⇒ Object

Runs the given work and asserts that the times gathered curve fit to match a power curve within a given error threshold.

Fit is calculated by #fit_power.

Ranges are specified by ::bench_range.

Eg:

def bench_algorithm
  assert_performance_power 0.9999 do |x|
    @obj.algorithm
  end
end



214
215
216
 
# File 'lib/minitest/benchmark.rb', line 214

def assert_performance_power threshold = 0.99, &work
  assert_performance validation_for_fit(:power, threshold), &work
end

#fit_error(xys) ⇒ Object

Takes an array of x/y pairs and calculates the general R^2 value.

See: en.wikipedia.org/wiki/Coefficient_of_determination



223
224
225
226
227
228
229
 
# File 'lib/minitest/benchmark.rb', line 223

def fit_error xys
  y_bar  = sigma(xys) { |x, y| y } / xys.size.to_f
  ss_tot = sigma(xys) { |x, y| (y    - y_bar) ** 2 }
  ss_err = sigma(xys) { |x, y| (yield(x) - y) ** 2 }

  1 - (ss_err / ss_tot)
end

#fit_exponential(xs, ys) ⇒ Object

To fit a functional form: y = ae^(bx).

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFittingExponential.html



238
239
240
241
242
243
244
245
246
247
248
249
250
251
 
# File 'lib/minitest/benchmark.rb', line 238

def fit_exponential xs, ys
  n     = xs.size
  xys   = xs.zip(ys)
  sxlny = sigma(xys) { |x,y| x * Math.log(y) }
  slny  = sigma(xys) { |x,y| Math.log(y)     }
  sx2   = sigma(xys) { |x,y| x * x           }
  sx    = sigma xs

  c = n * sx2 - sx ** 2
  a = (slny * sx2 - sx * sxlny) / c
  b = ( n * sxlny - sx * slny ) / c

  return Math.exp(a), b, fit_error(xys) { |x| Math.exp(a + b * x) }
end

#fit_linear(xs, ys) ⇒ Object

Fits the functional form: a + bx.

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFitting.html



283
284
285
286
287
288
289
290
291
292
293
294
295
296
 
# File 'lib/minitest/benchmark.rb', line 283

def fit_linear xs, ys
  n   = xs.size
  xys = xs.zip(ys)
  sx  = sigma xs
  sy  = sigma ys
  sx2 = sigma(xs)  { |x|   x ** 2 }
  sxy = sigma(xys) { |x,y| x * y  }

  c = n * sx2 - sx**2
  a = (sy * sx2 - sx * sxy) / c
  b = ( n * sxy - sx * sy ) / c

  return a, b, fit_error(xys) { |x| a + b * x }
end

#fit_logarithmic(xs, ys) ⇒ Object

To fit a functional form: y = a + b*ln(x).

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFittingLogarithmic.html



260
261
262
263
264
265
266
267
268
269
270
271
272
273
 
# File 'lib/minitest/benchmark.rb', line 260

def fit_logarithmic xs, ys
  n     = xs.size
  xys   = xs.zip(ys)
  slnx2 = sigma(xys) { |x,y| Math.log(x) ** 2 }
  slnx  = sigma(xys) { |x,y| Math.log(x)      }
  sylnx = sigma(xys) { |x,y| y * Math.log(x)  }
  sy    = sigma(xys) { |x,y| y                }

  c = n * slnx2 - slnx ** 2
  b = ( n * sylnx - sy * slnx ) / c
  a = (sy - b * slnx) / n

  return a, b, fit_error(xys) { |x| a + b * Math.log(x) }
end

#fit_power(xs, ys) ⇒ Object

To fit a functional form: y = ax^b.

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFittingPowerLaw.html



305
306
307
308
309
310
311
312
313
314
315
316
317
 
# File 'lib/minitest/benchmark.rb', line 305

def fit_power xs, ys
  n       = xs.size
  xys     = xs.zip(ys)
  slnxlny = sigma(xys) { |x, y| Math.log(x) * Math.log(y) }
  slnx    = sigma(xs)  { |x   | Math.log(x)               }
  slny    = sigma(ys)  { |   y| Math.log(y)               }
  slnx2   = sigma(xs)  { |x   | Math.log(x) ** 2          }

  b = (n * slnxlny - slnx * slny) / (n * slnx2 - slnx ** 2);
  a = (slny - b * slnx) / n

  return Math.exp(a), b, fit_error(xys) { |x| (Math.exp(a) * (x ** b)) }
end

#io ⇒ Object

:nodoc:



16
17
18
 
# File 'lib/minitest/benchmark.rb', line 16

def io # :nodoc:
  self.class.io
end

#sigma(enum, &block) ⇒ Object

Enumerates over enum mapping block if given, returning the sum of the result. Eg:

sigma([1, 2, 3])                # => 1 + 2 + 3 => 7
sigma([1, 2, 3]) { |n| n ** 2 } # => 1 + 4 + 9 => 14



326
327
328
329
 
# File 'lib/minitest/benchmark.rb', line 326

def sigma enum, &block
  enum = enum.map(&block) if block
  enum.inject { |sum, n| sum + n }
end

#validation_for_fit(msg, threshold) ⇒ Object

Returns a proc that calls the specified fit method and asserts that the error is within a tolerable threshold.



335
336
337
338
339
340
341
 
# File 'lib/minitest/benchmark.rb', line 335

def validation_for_fit msg, threshold
  proc do |range, times|
    a, b, rr = send "fit_#{msg}", range, times
    assert_operator rr, :>=, threshold
    [a, b, rr]
  end
end