Class: Vanity::Experiment::AbTest
- Defined in:
- lib/vanity/experiment/ab_test.rb
Overview
The meat.
Instance Attribute Summary
Attributes inherited from Base
#completed_at, #id, #name, #playground
Class Method Summary collapse
- .friendly_name ⇒ Object
-
.probability(score) ⇒ Object
Convert z-score to probability.
Instance Method Summary collapse
-
#alternative(value) ⇒ Object
Returns an Alternative with the specified value.
-
#alternatives(*args) ⇒ Object
Call this method once to set alternative values for this experiment (requires at least two values).
-
#choose ⇒ Object
Chooses a value for this experiment.
-
#chooses(value) ⇒ Object
Forces this experiment to use a particular alternative.
- #complete! ⇒ Object
-
#conclusion(score = score) ⇒ Object
Use the result of #score to derive a conclusion.
-
#destroy ⇒ Object
– Store/validate –.
-
#false_true ⇒ Object
(also: #true_false)
Defines an A/B test with two alternatives: false and true.
-
#fingerprint(alternative) ⇒ Object
Returns fingerprint (hash) for given alternative.
-
#initialize(*args) ⇒ AbTest
constructor
A new instance of AbTest.
-
#metrics(*args) ⇒ Object
Tells A/B test which metric we’re measuring, or returns metric in use.
-
#outcome ⇒ Object
Alternative chosen when this experiment completed.
-
#outcome_is(&block) ⇒ Object
Defines how the experiment can choose the optimal outcome on completion.
- #save ⇒ Object
-
#score(probability = 90) ⇒ Object
Scores alternatives based on the current tracking data.
-
#showing?(alternative) ⇒ Boolean
True if this alternative is currently showing (see #chooses).
-
#track!(metric_id, timestamp, count, *args) ⇒ Object
Called when tracking associated metric.
Methods inherited from Base
#active?, #complete_if, #created_at, #description, #identify, load, #type, type
Constructor Details
#initialize(*args) ⇒ AbTest
Returns a new instance of AbTest.
109 110 111 |
# File 'lib/vanity/experiment/ab_test.rb', line 109 def initialize(*args) super end |
Class Method Details
.friendly_name ⇒ Object
103 104 105 |
# File 'lib/vanity/experiment/ab_test.rb', line 103 def friendly_name "A/B Test" end |
Instance Method Details
#alternative(value) ⇒ Object
Returns an Alternative with the specified value.
168 169 170 |
# File 'lib/vanity/experiment/ab_test.rb', line 168 def alternative(value) alternatives.find { |alt| alt.value == value } end |
#alternatives(*args) ⇒ Object
Call this method once to set alternative values for this experiment (requires at least two values). Call without arguments to obtain current list of alternatives.
146 147 148 149 150 151 152 |
# File 'lib/vanity/experiment/ab_test.rb', line 146 def alternatives(*args) @alternatives = args.empty? ? [true, false] : args.clone class << self define_method :alternatives, instance_method(:_alternatives) end nil end |
#choose ⇒ Object
Chooses a value for this experiment. You probably want to use the Rails helper method ab_test instead.
This method picks an alternative for the current identity and returns the alternative’s value. It will consistently choose the same alternative for the same identity, and randomly split alternatives between different identities.
197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 |
# File 'lib/vanity/experiment/ab_test.rb', line 197 def choose if @playground.collecting? if active? identity = identity() index = connection.ab_showing(@id, identity) unless index index = alternative_for(identity) if !@playground.using_js? connection.ab_add_participant @id, index, identity check_completion! end end else index = connection.ab_get_outcome(@id) || alternative_for(identity) end else identity = identity() @showing ||= {} @showing[identity] ||= alternative_for(identity) index = @showing[identity] end alternatives[index.to_i] end |
#chooses(value) ⇒ Object
Forces this experiment to use a particular alternative. You’ll want to use this from your test cases to test for the different alternatives.
247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 |
# File 'lib/vanity/experiment/ab_test.rb', line 247 def chooses(value) if @playground.collecting? if value.nil? connection.ab_not_showing @id, identity else index = @alternatives.index(value) #add them to the experiment unless they are already in it unless index == connection.ab_showing(@id, identity) connection.ab_add_participant @id, index, identity check_completion! end raise ArgumentError, "No alternative #{value.inspect} for #{name}" unless index if (connection.ab_showing(@id, identity) && connection.ab_showing(@id, identity) != index) || alternative_for(identity) != index connection.ab_show @id, identity, index end end else @showing ||= {} @showing[identity] = value.nil? ? nil : @alternatives.index(value) end self end |
#complete! ⇒ Object
401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 |
# File 'lib/vanity/experiment/ab_test.rb', line 401 def complete! return unless @playground.collecting? && active? super if @outcome_is begin result = @outcome_is.call outcome = result.id if Alternative === result && result.experiment == self rescue warn "Error in AbTest#complete!: #{$!}" end else best = score.best outcome = best.id if best end # TODO: logging connection.ab_set_outcome @id, outcome || 0 end |
#conclusion(score = score) ⇒ Object
Use the result of #score to derive a conclusion. Returns an array of claims.
331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 |
# File 'lib/vanity/experiment/ab_test.rb', line 331 def conclusion(score = score) claims = [] participants = score.alts.inject(0) { |t,alt| t + alt.participants } claims << case participants when 0 ; "There are no participants in this experiment yet." when 1 ; "There is one participant in this experiment." else ; "There are #{participants} participants in this experiment." end # only interested in sorted alternatives with conversion sorted = score.alts.select { |alt| alt.measure > 0.0 }.sort_by(&:measure).reverse if sorted.size > 1 # start with alternatives that have conversion, from best to worst, # then alternatives with no conversion. sorted |= score.alts # we want a result that's clearly better than 2nd best. best, second = sorted[0], sorted[1] if best.measure > second.measure diff = ((best.measure - second.measure) / second.measure * 100).round better = " (%d%% better than %s)" % [diff, second.name] if diff > 0 claims << "The best choice is %s: it converted at %.1f%%%s." % [best.name, best.measure * 100, better] if best.probability >= 90 claims << "With %d%% probability this result is statistically significant." % score.best.probability else claims << "This result is not statistically significant, suggest you continue this experiment." end sorted.delete best end sorted.each do |alt| if alt.measure > 0.0 claims << "%s converted at %.1f%%." % [alt.name.gsub(/^o/, "O"), alt.measure * 100] else claims << "%s did not convert." % alt.name.gsub(/^o/, "O") end end else claims << "This experiment did not run long enough to find a clear winner." end claims << "#{score.choice.name.gsub(/^o/, "O")} selected as the best alternative." if score.choice claims end |
#destroy ⇒ Object
– Store/validate –
422 423 424 425 |
# File 'lib/vanity/experiment/ab_test.rb', line 422 def destroy connection.destroy_experiment @id super end |
#false_true ⇒ Object Also known as: true_false
Defines an A/B test with two alternatives: false and true. This is the default pair of alternatives, so just syntactic sugar for those who love being explicit.
182 183 184 |
# File 'lib/vanity/experiment/ab_test.rb', line 182 def false_true alternatives false, true end |
#fingerprint(alternative) ⇒ Object
Returns fingerprint (hash) for given alternative. Can be used to lookup alternative for experiment without revealing what values are available (e.g. choosing alternative from HTTP query parameter).
224 225 226 |
# File 'lib/vanity/experiment/ab_test.rb', line 224 def fingerprint(alternative) Digest::MD5.hexdigest("#{id}:#{alternative.id}")[-10,10] end |
#metrics(*args) ⇒ Object
Tells A/B test which metric we’re measuring, or returns metric in use.
125 126 127 128 |
# File 'lib/vanity/experiment/ab_test.rb', line 125 def metrics(*args) @metrics = args.map { |id| @playground.metric(id) } unless args.empty? @metrics end |
#outcome ⇒ Object
Alternative chosen when this experiment completed.
395 396 397 398 399 |
# File 'lib/vanity/experiment/ab_test.rb', line 395 def outcome return unless @playground.collecting? outcome = connection.ab_get_outcome(@id) outcome && _alternatives[outcome] end |
#outcome_is(&block) ⇒ Object
Defines how the experiment can choose the optimal outcome on completion.
By default, Vanity will take the best alternative (highest conversion rate) and use that as the outcome. You experiment may have different needs, maybe you want the least performing alternative, or factor cost in the equation?
The default implementation reads like this:
outcome_is do
a, b = alternatives
# a is expensive, only choose a if it performs 2x better than b
a.measure > b.measure * 2 ? a : b
end
388 389 390 391 392 |
# File 'lib/vanity/experiment/ab_test.rb', line 388 def outcome_is(&block) raise ArgumentError, "Missing block" unless block raise "outcome_is already called on this experiment" if @outcome_is @outcome_is = block end |
#save ⇒ Object
427 428 429 430 431 432 433 434 435 436 437 438 439 |
# File 'lib/vanity/experiment/ab_test.rb', line 427 def save true_false unless @alternatives fail "Experiment #{name} needs at least two alternatives" unless @alternatives.size >= 2 super if @metrics.nil? || @metrics.empty? warn "Please use metrics method to explicitly state which metric you are measuring against." metric = @playground.metrics[id] ||= Vanity::Metric.new(@playground, name) @metrics = [metric] end @metrics.each do |metric| metric.hook &method(:track!) end end |
#score(probability = 90) ⇒ Object
Scores alternatives based on the current tracking data. This method returns a structure with the following attributes:
- :alts
-
Ordered list of alternatives, populated with scoring info.
- :base
-
Second best performing alternative.
- :least
-
Least performing alternative (but more than zero conversion).
- :choice
-
Choice alterntive, either the outcome or best alternative.
Alternatives returned by this method are populated with the following attributes:
- :z_score
-
Z-score (relative to the base alternative).
- :probability
-
Probability (z-score mapped to 0, 90, 95, 99 or 99.9%).
- :difference
-
Difference from the least performant altenative.
The choice alternative is set only if its probability is higher or equal to the specified probability (default is 90%).
300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 |
# File 'lib/vanity/experiment/ab_test.rb', line 300 def score(probability = 90) alts = alternatives # sort by conversion rate to find second best and 2nd best sorted = alts.sort_by(&:measure) base = sorted[-2] # calculate z-score pc = base.measure nc = base.participants alts.each do |alt| p = alt.measure n = alt.participants alt.z_score = (p - pc) / ((p * (1-p)/n) + (pc * (1-pc)/nc)).abs ** 0.5 alt.probability = AbTest.probability(alt.z_score) end # difference is measured from least performant if least = sorted.find { |alt| alt.measure > 0 } alts.each do |alt| if alt.measure > least.measure alt.difference = (alt.measure - least.measure) / least.measure * 100 end end end # best alternative is one with highest conversion rate (best shot). # choice alternative can only pick best if we have high probability (>90%). best = sorted.last if sorted.last.measure > 0.0 choice = outcome ? alts[outcome.id] : (best && best.probability >= probability ? best : nil) Struct.new(:alts, :best, :base, :least, :choice).new(alts, best, base, least, choice) end |
#showing?(alternative) ⇒ Boolean
True if this alternative is currently showing (see #chooses).
272 273 274 275 276 277 278 279 280 |
# File 'lib/vanity/experiment/ab_test.rb', line 272 def showing?(alternative) identity = identity() if @playground.collecting? (connection.ab_showing(@id, identity) || alternative_for(identity)) == alternative.id else @showing ||= {} @showing[identity] == alternative.id end end |
#track!(metric_id, timestamp, count, *args) ⇒ Object
Called when tracking associated metric.
442 443 444 445 446 447 448 449 450 451 |
# File 'lib/vanity/experiment/ab_test.rb', line 442 def track!(metric_id, , count, *args) return unless active? identity = identity() rescue nil if identity return if connection.ab_showing(@id, identity) index = alternative_for(identity) connection.ab_add_conversion @id, index, identity, count check_completion! end end |