Class: JmeterPerf::Report::Comparator

Inherits:
Object
  • Object
show all
Defined in:
lib/jmeter_perf/report/comparator.rb

Overview

Comparator performs statistical comparison between two performance reports. It calculates metrics like Cohen’s D and T-statistic to measure the effect size and generates comparison reports in various formats.

Constant Summary collapse

EFFECT_SIZE_LIMITS =

Effect size thresholds according to Sawilowsky’s rule of thumb

{
  vsmall: 0.01,  # very small
  small: 0.2,    # small
  medium: 0.5,   # medium
  large: 0.8,    # large
  vlarge: 1.2,   # very large
  huge: 2.0      # huge
}
COMPARISON_REPORT_HEADER = 
[
  "Label",
  "Total Requests",
  "Total Elapsed Time",
  "RPM",
  "Errors",
  "Error %",
  "Min",
  "Max",
  "Avg",
  "SD",
  "P10",
  "P50",
  "P95"
]

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(base_report, test_report, name = nil) ⇒ Comparator

Initializes a Comparator instance to compare two reports.

Parameters:

  • base_report (Summary) —

    the base performance report

  • test_report (Summary) —

    the test performance report

  • name (String, nil) (defaults to: nil) —

    an optional name for the comparison (default: nil)



49
50
51
52
53
54
 
# File 'lib/jmeter_perf/report/comparator.rb', line 49

def initialize(base_report, test_report, name = nil)
  @base_report = base_report
  @test_report = test_report
  @name = name&.gsub(/\s+/, "_")
  compare_reports!
end

Instance Attribute Details

#cohens_d ⇒ Float (readonly)

Returns the calculated Cohen’s D value.

Returns:

  • (Float) —

    the calculated Cohen’s D value

See Also:



9
10
11
 
# File 'lib/jmeter_perf/report/comparator.rb', line 9

def cohens_d
  @cohens_d
end

#human_rating ⇒ String (readonly)

Returns a human-readable rating of the comparison result.

Returns:

  • (String) —

    a human-readable rating of the comparison result



14
15
16
 
# File 'lib/jmeter_perf/report/comparator.rb', line 14

def human_rating
  @human_rating
end

#name ⇒ String (readonly)

Returns the name of the comparison, if provided.

Returns:

  • (String) —

    the name of the comparison, if provided



16
17
18
 
# File 'lib/jmeter_perf/report/comparator.rb', line 16

def name
  @name
end

#t_statistic ⇒ Float (readonly)

Returns the calculated T-statistic value.

Returns:

  • (Float) —

    the calculated T-statistic value

See Also:



12
13
14
 
# File 'lib/jmeter_perf/report/comparator.rb', line 12

def t_statistic
  @t_statistic
end

Instance Method Details

#generate_reports(output_dir: ".", output_format: :all) ⇒ void

This method returns an undefined value.

Generates comparison reports in specified formats.

Parameters:

  • output_dir (String) (defaults to: ".") —

    the directory for output files (default: “.”)

  • output_format (Symbol) (defaults to: :all) —

    the format for the report, e.g., :html, :csv, :stdout (default: :all)

Raises:

  • (ArgumentError) —

    if the output format is invalid



76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
 
# File 'lib/jmeter_perf/report/comparator.rb', line 76

def generate_reports(output_dir: ".", output_format: :all)
  case output_format
  when :all
    generate_html_report(File.join(output_dir, "#{@name}_comparison_report.html"))
    generate_csv_report(File.join(output_dir, "#{@name}_comparison_report.csv"))
    print_comparison
  when :html
    generate_html_report(File.join(output_dir, "#{@name}_comparison_report.html"))
  when :csv
    generate_csv_report(File.join(output_dir, "#{@name}_comparison_report.csv"))
  when :stdout
    print_comparison
  else
    raise ArgumentError, "Invalid output format: #{output_format}"
  end
end

#pass?(cohens_d_limit: nil, effect_size: :vsmall) ⇒ Boolean

Note:

If no Cohen’s D limit is provided, the ‘effect_size` threshold is used.

Note:

Positive effect size indicates an increase in performance and is considered a pass.

Checks if the comparison passes based on Cohen’s D and effect size threshold.

Parameters:

  • cohens_d_limit (Float, nil) (defaults to: nil) —

    optional negative limit for Cohen’s D

  • effect_size (Symbol) (defaults to: :vsmall) —

    the desired effect size threshold (default: :vsmall). See EFFECT_SIZE_LIMITS for options.

Returns:

  • (Boolean) —

    true if comparison meets the criteria

Raises:

  • (ArgumentError) —

    if the effect size is invalid



64
65
66
67
68
 
# File 'lib/jmeter_perf/report/comparator.rb', line 64

def pass?(cohens_d_limit: nil, effect_size: :vsmall)
  limit = cohens_d_limit || EFFECT_SIZE_LIMITS[effect_size]
  raise ArgumentError, "Invalid effect size: #{effect_size}" unless limit
  cohens_d >= -limit.abs
end

#print_comparison ⇒ Object 



126
127
128
 
# File 'lib/jmeter_perf/report/comparator.rb', line 126

def print_comparison
  puts self
end

#to_s ⇒ Object 



93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
 
# File 'lib/jmeter_perf/report/comparator.rb', line 93

def to_s
  report_text = "Comparison Report\n"
  report_text << "Cohen's D: #{@cohens_d}\n"
  report_text << "Human Rating: #{@human_rating}\n"
  report_text << "-" * 135 + "\n"

  header_format = "%-15s %-17s %-18s %-8s %-8s %-9s %-7s %-7s %-8s %-8s %-8s %-8s %-8s\n"
  row_format = "%-15s %-17d %-18d %-8.2f %-8d %-9.2f %-7d %-7d %-8.2f %-8.2f %-8.2f %-8.2f %-8.2f\n"

  report_text << sprintf(header_format, *COMPARISON_REPORT_HEADER)
  report_text << "-" * 135 + "\n"

  [@base_report, @test_report].each_with_index do |report, index|
    report_text << sprintf(row_format,
      (index == 0) ? "Base Metric" : "Test Metric",
      report.total_requests,
      report.total_run_time,
      report.rpm,
      report.total_errors,
      report.error_percentage,
      report.min,
      report.max,
      report.avg,
      report.std,
      report.p10,
      report.p50,
      report.p95)
  end

  report_text << "-" * 135 + "\n"
  report_text
end