Class: Benchmark::IPS::Job
- Inherits:
-
Object
- Object
- Benchmark::IPS::Job
- Defined in:
- lib/benchmark/ips/job.rb,
lib/benchmark/ips/job/entry.rb,
lib/benchmark/ips/job/noop_report.rb,
lib/benchmark/ips/job/stdout_report.rb
Overview
Benchmark jobs.
Defined Under Namespace
Classes: Entry, NoopReport, StdoutReport
Constant Summary collapse
- MICROSECONDS_PER_100MS =
Microseconds per 100 millisecond.
100_000
- MICROSECONDS_PER_SECOND =
Microseconds per second.
Timing::MICROSECONDS_PER_SECOND
- MAX_TIME_SKEW =
The percentage of the expected runtime to allow before reporting a weird runtime
0.05
- POW_2_30 =
1 << 30
Instance Attribute Summary collapse
-
#compare ⇒ Boolean
readonly
Determining whether to run comparison utility.
-
#confidence ⇒ Integer
Confidence.
-
#full_report ⇒ Report
readonly
Report object containing information about the run.
-
#hold ⇒ Boolean
Determining whether to hold results between Ruby invocations.
-
#iterations ⇒ Integer
Warmup and calculation iterations.
-
#list ⇒ Array<Entry>
readonly
Two-element arrays, consisting of label and block pairs.
-
#quiet ⇒ Boolean
Silence output.
-
#stats ⇒ Object
Statistics model.
-
#suite ⇒ Benchmark::IPS::NoopSuite
Suite.
-
#time ⇒ Integer
Calculation time setter and getter (in seconds).
-
#timing ⇒ Hash
readonly
Storing Iterations in time period.
-
#warmup ⇒ Integer
Warmup time setter and getter (in seconds).
Instance Method Summary collapse
- #all_results_have_been_run? ⇒ Boolean
- #clear_held_results ⇒ Object
-
#compare!(order: :fastest) ⇒ Object
Run comparison utility.
-
#compare? ⇒ Boolean
Return true if job needs to be compared.
-
#config(opts) ⇒ Object
Job configuration options, set @warmup and @time.
-
#create_report(label, measured_us, iter, samples, cycles) ⇒ Report::Entry
Create report by add entry to @full_report.
- #create_stats(samples) ⇒ Object
-
#cycles_per_100ms(time_msec, iters) ⇒ Integer
Calculate the cycles needed to run for approx 100ms, given the number of iterations to run the given time.
-
#generate_json ⇒ Object
Generate json from @full_report.
-
#hold!(held_path) ⇒ Object
Hold after each iteration.
-
#hold? ⇒ Boolean
Return true if results are held while multiple Ruby invocations.
-
#initialize(opts = {}) ⇒ Job
constructor
Instantiate the Benchmark::IPS::Job.
-
#item(label = "", str = nil, &blk) ⇒ Object
(also: #report)
Registers the given label and block pair in the job list.
-
#iterations_per_sec(cycles, time_us) ⇒ Float
Calculate the interations per second given the number of cycles run and the time in microseconds that elapsed.
-
#json!(path = "data.json") ⇒ Object
Generate json to given path, defaults to “data.json”.
-
#json? ⇒ Boolean
Return true if job needs to generate json.
- #load_held_results ⇒ Object
- #reporter(quiet:) ⇒ Object
- #run ⇒ Object
-
#run_benchmark ⇒ Object
Run calculation.
-
#run_comparison ⇒ Object
Run comparison of entries in @full_report.
-
#run_single? ⇒ Boolean
Return true if items are to be run one at a time.
-
#run_warmup ⇒ Object
Run warmup.
-
#save!(held_path) ⇒ Object
Save interim results.
- #save_held_results ⇒ Object
-
#time_us(before, after) ⇒ Float
Calculate the time difference of before and after in microseconds.
Constructor Details
#initialize(opts = {}) ⇒ Job
Instantiate the Benchmark::IPS::Job.
63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 |
# File 'lib/benchmark/ips/job.rb', line 63 def initialize opts={} @list = [] @run_single = false @json_path = false @compare = false @compare_order = :fastest @held_path = nil @held_results = nil @timing = Hash.new 1 # default to 1 in case warmup isn't run @full_report = Report.new # Default warmup and calculation time in seconds. @warmup = 2 @time = 5 @iterations = 1 # Default statistical model @stats = :sd @confidence = 95 self.quiet = false end |
Instance Attribute Details
#compare ⇒ Boolean (readonly)
Determining whether to run comparison utility.
20 21 22 |
# File 'lib/benchmark/ips/job.rb', line 20 def compare @compare end |
#confidence ⇒ Integer
Confidence.
52 53 54 |
# File 'lib/benchmark/ips/job.rb', line 52 def confidence @confidence end |
#full_report ⇒ Report (readonly)
Report object containing information about the run.
28 29 30 |
# File 'lib/benchmark/ips/job.rb', line 28 def full_report @full_report end |
#hold ⇒ Boolean
Determining whether to hold results between Ruby invocations
24 25 26 |
# File 'lib/benchmark/ips/job.rb', line 24 def hold @hold end |
#iterations ⇒ Integer
Warmup and calculation iterations.
44 45 46 |
# File 'lib/benchmark/ips/job.rb', line 44 def iterations @iterations end |
#list ⇒ Array<Entry> (readonly)
Two-element arrays, consisting of label and block pairs.
16 17 18 |
# File 'lib/benchmark/ips/job.rb', line 16 def list @list end |
#quiet ⇒ Boolean
Silence output
56 57 58 |
# File 'lib/benchmark/ips/job.rb', line 56 def quiet @quiet end |
#stats ⇒ Object
Statistics model.
48 49 50 |
# File 'lib/benchmark/ips/job.rb', line 48 def stats @stats end |
#suite ⇒ Benchmark::IPS::NoopSuite
Suite
60 61 62 |
# File 'lib/benchmark/ips/job.rb', line 60 def suite @suite end |
#time ⇒ Integer
Calculation time setter and getter (in seconds).
40 41 42 |
# File 'lib/benchmark/ips/job.rb', line 40 def time @time end |
#timing ⇒ Hash (readonly)
Storing Iterations in time period.
32 33 34 |
# File 'lib/benchmark/ips/job.rb', line 32 def timing @timing end |
#warmup ⇒ Integer
Warmup time setter and getter (in seconds).
36 37 38 |
# File 'lib/benchmark/ips/job.rb', line 36 def warmup @warmup end |
Instance Method Details
#all_results_have_been_run? ⇒ Boolean
238 239 240 |
# File 'lib/benchmark/ips/job.rb', line 238 def all_results_have_been_run? @full_report.entries.size == @list.size end |
#clear_held_results ⇒ Object
242 243 244 |
# File 'lib/benchmark/ips/job.rb', line 242 def clear_held_results File.delete @held_path if File.exist?(@held_path) end |
#compare!(order: :fastest) ⇒ Object
Run comparison utility.
121 122 123 124 |
# File 'lib/benchmark/ips/job.rb', line 121 def compare!(order: :fastest) @compare = true @compare_order = order end |
#compare? ⇒ Boolean
Return true if job needs to be compared.
116 117 118 |
# File 'lib/benchmark/ips/job.rb', line 116 def compare? @compare end |
#config(opts) ⇒ Object
Job configuration options, set @warmup and @time.
91 92 93 94 95 96 97 98 99 100 |
# File 'lib/benchmark/ips/job.rb', line 91 def config opts @warmup = opts[:warmup] if opts[:warmup] @time = opts[:time] if opts[:time] @suite = opts[:suite] if opts[:suite] @iterations = opts[:iterations] if opts[:iterations] @stats = opts[:stats] if opts[:stats] @confidence = opts[:confidence] if opts[:confidence] self.quiet = opts[:quiet] if opts.key?(:quiet) self.suite = opts[:suite] end |
#create_report(label, measured_us, iter, samples, cycles) ⇒ Report::Entry
Create report by add entry to @full_report.
393 394 395 |
# File 'lib/benchmark/ips/job.rb', line 393 def create_report(label, measured_us, iter, samples, cycles) @full_report.add_entry label, measured_us, iter, samples, cycles end |
#create_stats(samples) ⇒ Object
365 366 367 368 369 370 371 372 373 374 |
# File 'lib/benchmark/ips/job.rb', line 365 def create_stats(samples) case @stats when :sd Stats::SD.new(samples) when :bootstrap Stats::Bootstrap.new(samples, @confidence) else raise "unknown stats #{@stats}" end end |
#cycles_per_100ms(time_msec, iters) ⇒ Integer
Calculate the cycles needed to run for approx 100ms, given the number of iterations to run the given time.
190 191 192 193 |
# File 'lib/benchmark/ips/job.rb', line 190 def cycles_per_100ms time_msec, iters cycles = ((MICROSECONDS_PER_100MS / time_msec) * iters).to_i cycles <= 0 ? 1 : cycles end |
#generate_json ⇒ Object
Generate json from @full_report.
382 383 384 |
# File 'lib/benchmark/ips/job.rb', line 382 def generate_json @full_report.generate_json @json_path if json? end |
#hold!(held_path) ⇒ Object
Hold after each iteration.
134 135 136 137 |
# File 'lib/benchmark/ips/job.rb', line 134 def hold!(held_path) @held_path = held_path @run_single = true end |
#hold? ⇒ Boolean
Return true if results are held while multiple Ruby invocations
128 129 130 |
# File 'lib/benchmark/ips/job.rb', line 128 def hold? !!@held_path end |
#item(label = "", str = nil, &blk) ⇒ Object Also known as: report
Registers the given label and block pair in the job list.
172 173 174 175 176 177 178 179 180 181 182 |
# File 'lib/benchmark/ips/job.rb', line 172 def item(label="", str=nil, &blk) # :yield: if blk and str raise ArgumentError, "specify a block and a str, but not both" end action = str || blk raise ArgumentError, "no block or string" unless action @list.push Entry.new(label, action) self end |
#iterations_per_sec(cycles, time_us) ⇒ Float
Calculate the interations per second given the number of cycles run and the time in microseconds that elapsed.
208 209 210 |
# File 'lib/benchmark/ips/job.rb', line 208 def iterations_per_sec cycles, time_us MICROSECONDS_PER_SECOND * (cycles.to_f / time_us.to_f) end |
#json!(path = "data.json") ⇒ Object
Generate json to given path, defaults to “data.json”.
162 163 164 |
# File 'lib/benchmark/ips/job.rb', line 162 def json!(path="data.json") @json_path = path end |
#json? ⇒ Boolean
Return true if job needs to generate json.
157 158 159 |
# File 'lib/benchmark/ips/job.rb', line 157 def json? !!@json_path end |
#load_held_results ⇒ Object
212 213 214 215 216 217 218 219 220 221 |
# File 'lib/benchmark/ips/job.rb', line 212 def load_held_results return unless @held_path && File.exist?(@held_path) && !File.zero?(@held_path) require "json" @held_results = {} JSON.load(IO.read(@held_path)).each do |result| @held_results[result['item']] = result create_report(result['item'], result['measured_us'], result['iter'], create_stats(result['samples']), result['cycles']) end end |
#reporter(quiet:) ⇒ Object
110 111 112 |
# File 'lib/benchmark/ips/job.rb', line 110 def reporter(quiet:) quiet ? NoopReport.new : StdoutReport.new end |
#run ⇒ Object
246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 |
# File 'lib/benchmark/ips/job.rb', line 246 def run if @warmup && @warmup != 0 then @stdout.start_warming @iterations.times do run_warmup end end @stdout.start_running @iterations.times do |n| run_benchmark end @stdout. end |
#run_benchmark ⇒ Object
Run calculation.
311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 |
# File 'lib/benchmark/ips/job.rb', line 311 def run_benchmark @list.each do |item| next if run_single? && @held_results && @held_results.key?(item.label) @suite.running item.label, @time @stdout.running item.label, @time Timing.clean_env iter = 0 measurements_us = [] # Running this number of cycles should take around 100ms. cycles = @timing[item] target = Timing.add_second Timing.now, @time begin before = Timing.now item.call_times cycles after = Timing.now # If for some reason the timing said this took no time (O_o) # then ignore the iteration entirely and start another. iter_us = Timing.time_us before, after next if iter_us <= 0.0 iter += cycles measurements_us << iter_us end while Timing.now < target final_time = before measured_us = measurements_us.inject(:+) samples = measurements_us.map { |time_us| iterations_per_sec cycles, time_us } rep = create_report(item.label, measured_us, iter, create_stats(samples), cycles) if (final_time - target).abs >= (@time.to_f * MAX_TIME_SKEW) rep.show_total_time! end @stdout.add_report rep, caller(1).first @suite.add_report rep, caller(1).first break if run_single? end end |
#run_comparison ⇒ Object
Run comparison of entries in @full_report.
377 378 379 |
# File 'lib/benchmark/ips/job.rb', line 377 def run_comparison @full_report.run_comparison(@compare_order) if compare? end |
#run_single? ⇒ Boolean
Return true if items are to be run one at a time. For the traditional hold, this is true
151 152 153 |
# File 'lib/benchmark/ips/job.rb', line 151 def run_single? @run_single end |
#run_warmup ⇒ Object
Run warmup.
264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 |
# File 'lib/benchmark/ips/job.rb', line 264 def run_warmup @list.each do |item| next if run_single? && @held_results && @held_results.key?(item.label) @suite.warming item.label, @warmup @stdout.warming item.label, @warmup Timing.clean_env # Run for up to half of the configured warmup time with an increasing # number of cycles to reduce overhead and improve accuracy. # This also avoids running with a constant number of cycles, which a # JIT might speculate on and then have to recompile in #run_benchmark. before = Timing.now target = Timing.add_second before, @warmup / 2.0 cycles = 1 begin t0 = Timing.now item.call_times cycles t1 = Timing.now warmup_iter = cycles warmup_time_us = Timing.time_us(t0, t1) # If the number of cycles would go outside the 32-bit signed integers range # then exit the loop to avoid overflows and start the 100ms warmup runs break if cycles >= POW_2_30 cycles *= 2 end while Timing.now + warmup_time_us * 2 < target cycles = cycles_per_100ms warmup_time_us, warmup_iter @timing[item] = cycles # Run for the remaining of warmup in a similar way as #run_benchmark. target = Timing.add_second before, @warmup while Timing.now + MICROSECONDS_PER_100MS < target item.call_times cycles end @stdout.warmup_stats warmup_time_us, @timing[item] @suite.warmup_stats warmup_time_us, @timing[item] break if run_single? end end |
#save!(held_path) ⇒ Object
Save interim results. Similar to hold, but all reports are run The report label must change for each invocation. One way to achieve this is to include the version in the label.
143 144 145 146 |
# File 'lib/benchmark/ips/job.rb', line 143 def save!(held_path) @held_path = held_path @run_single = false end |
#save_held_results ⇒ Object
223 224 225 226 227 228 229 230 231 232 233 234 235 236 |
# File 'lib/benchmark/ips/job.rb', line 223 def save_held_results return unless @held_path require "json" data = full_report.entries.map { |e| { 'item' => e.label, 'measured_us' => e.microseconds, 'iter' => e.iterations, 'samples' => e.samples, 'cycles' => e.measurement_cycle } } IO.write(@held_path, JSON.generate(data) << "\n") end |
#time_us(before, after) ⇒ Float
Calculate the time difference of before and after in microseconds.
199 200 201 |
# File 'lib/benchmark/ips/job.rb', line 199 def time_us before, after (after.to_f - before.to_f) * MICROSECONDS_PER_SECOND end |