GitLab Experiment

Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it.

This library provides a clean and elegant DSL (domain specific language) to define, run, and track your GitLab experiment.

When we discuss the behavior of this gem, we'll use terms like experiment, context, control, candidate, and variant. It's worth defining these terms so they're more understood.

• experiment is any deviation of code paths we want to run sometimes and not others.
• context is used to identify a consistent experience we'll provide in an experiment.
• control is the default, or "original" code path.
• candidate defines that there's one experimental code path.
• variant(s) is used when more than one experimental code path exists.

Candidate and variant are the same concept, but simplify how we speak about experimental paths.

[[TOC]]

Installation

Add the gem to your Gemfile and then bundle install.

gem 'gitlab-experiment'

If you're using Rails, you can install the initializer which provides basic configuration, documentation, and the base experiment class that all your experiments can inherit from.

$ rails generate gitlab:experiment:install

Implementing an experiment

For the sake of an example let's make one up. Let's run an experiment on what we render for disabling desktop notifications.

In our control (current world) we show a simple toggle interface labeled, "Notifications." In our experiment we want a "Turn on/off desktop notifications" button with a confirmation.

The behavior will be the same, but the interface will be different and may involve more or fewer steps.

Our hypothesis is that this will make the action more clear and will help in making a choice about if that's what the user really wants to do.

We'll name our experiment notification_toggle. This name is prefixed based on configuration. If you've set config.name_prefix = 'gitlab', the experiment name would be gitlab_notification_toggle elsewhere.

When you implement an experiment you'll need to provide a name, and a context. The name can show up in tracking calls, and potentially other aspects. The context determines the variant assigned, and should be consistent between calls. We'll discuss migrating context in later examples.

A context "key" represents the unique id of a context. It allows us to give the same experience between different calls to the experiment and can be used in caching.

Now in our experiment we're going to render one of two views: the control will be our current view, and the candidate will be the new toggle button with a confirmation flow.

class SubscriptionsController < ApplicationController
  def show
    experiment(:notification_toggle, actor: user) do |e|
      e.use { render_toggle } # control
      e.try { render_button } # candidate
    end
  end
end

You can define the experiment using simple control/candidate paths, or provide named variants.

Handling multi-variant experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.

experiment(:notification_toggle, actor: user) do |e|
  e.use { render_toggle } # control
  e.try(:variant_one) { render_button(confirmation: true) }
  e.try(:variant_two) { render_button(confirmation: false) }
end

Understanding how an experiment can change behavior is important in evaluating its performance.

To this end, we track events that are important by calling the same experiment elsewhere in code. By using the same context, you'll have consistent behavior and the ability to track events to it.

experiment(:notification_toggle, actor: user).track(:clicked_button)

Custom experiments

You can craft more advanced behaviors by defining custom experiments at a higher level. To do this you can define a class that inherits from ApplicationExperiment (or Gitlab::Experiment).

Let's say you want to do more advanced segmentation, or provide default behavior for the variants on the experiment we've already outlined above -- that way if the variants aren't defined in the block at the time the experiment is run, these methods will be used.

You can generate a custom experiment by running:

$ rails generate gitlab:experiment NotificationToggle control candidate

This will generate a file in app/experiments/notification_toggle_experiment.rb, as well as a test file for you to further expand on.

Here are some examples of what you can introduce once you have a custom experiment defined.

class NotificationToggleExperiment < ApplicationExperiment
  # Segment any account less than 2 weeks old into the candidate, without
  # asking the variant resolver to decide which variant to provide.
  segment :account_age, variant: :candidate

  # Define the default control behavior, which can be overridden at
  # experiment time.
  def control_behavior
    render_toggle
  end

  # Define the default candidate behavior, which can be overridden
  # at experiment time.
  def candidate_behavior
    render_button
  end

  private

  def account_age
    context.actor && context.actor.created_at < 2.weeks.ago
  end
end

# The class will be looked up based on the experiment name provided. 
exp = experiment(:notification_toggle, actor: user)
exp # => instance of NotificationToggleExperiment

# Run the experiment -- returning the result.
exp.run

# Track an event on the experiment we've defined.
exp.track(:clicked_button)

You can now also do things very similar to the simple examples and override the default variant behaviors defined in the custom experiment -- keeping in mind that this should be carefully considered within the scope of your experiment.

experiment(:notification_toggle, actor: user) do |e|
  e.use { render_special_toggle } # override default control behavior
end

You can also use the lower level class interface...

### Using the `.run` approach This is useful if you haven't included the DSL and so don't have access to the `experiment` method, but still want to execute an experiment. This is ultimately what the `experiment` method calls through to, and the method signatures are the same. ```ruby exp = Gitlab::Experiment.run(:notification_toggle, actor: user) do |e| # Context may be passed in the block, but must be finalized before calling # run or track. e.context(project: project) # add the project to the context # Define the control and candidate variant. e.use { render_toggle } # control e.try { render_button } # candidate end # Track an event on the experiment we've defined. exp.track(:clicked_button) ```

You can also specify the variant to use for segmentation...

### Specifying variant Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment. It's important to know what this might do to your data during rollout, so use this with careful consideration. ```ruby experiment(:notification_toggle, :no_interface, actor: user) do |e| e.use { render_toggle } # control e.try { render_button } # candidate e.try(:no_interface) { no_interface! } # variant end ``` Or you can set the variant within the block. This allows using unique segmentation logic or variant resolution if you need it. ```ruby experiment(:notification_toggle, actor: user) do |e| # Variant selection must be done before calling run or track. e.variant(:no_interface) # set the variant # ... end ``` Or it can be specified in the call to run if you call it from within the block. ```ruby experiment(:notification_toggle, actor: user) do |e| # ... # Variant selection can be specified when calling run. e.run(:no_interface) end ```

Segmentation rules

This library comes with the capability to segment contexts into a specific variant, before asking the variant resolver which variant to provide.

Segmentation can be achieved by using a custom experiment class and specifying the segmentation rules at a class level.

class NotificationToggleExperiment < ApplicationExperiment
  segment(variant: :variant_one) { context.actor.username == 'jejacks0n' }
  segment(variant: :variant_two) { context.actor.created_at < 2.weeks.ago }
end

In the previous examples, any user with the username 'jejacks0n' would always receive the experience defined in "variant_one". As well, any account less than 2 weeks old would get the alternate experience defined in "variant_two".

When an experiment is run, the segmentation rules are executed in the order they're defined. The first segmentation rule to produce a truthy result is the one which gets used to assign the variant. The remaining segmentation rules are skipped.

This means that any user with the name 'jejacks0n', regardless of account age, will always be provided the experience as defined in "variant_one".

Return value

By default the return value is a Gitlab::Experiment instance. In simple cases you may want only the results of the experiment though. You can call run within the block to get the return value of the assigned variant.

experiment(:notification_toggle) do |e|
  e.use { 'A' }
  e.try { 'B' }
  e.run
end # => 'A'

Including the DSL

By default, Gitlab::Experiment injects itself into the controller and view layers. This exposes the experiment method application wide in those layers.

Some experiments may extend outside of those layers, so you may want to include it elsewhere. For instance in a mailer, service object, background job, or similar.

Note: In a lot of these contexts you may not have a reference to the request (unless you pass it in, or provide access to it) which may be needed if you want to enable cookie behaviors and track that through to user conversion.

class WelcomeMailer < ApplicationMailer
  include Gitlab::Experiment::Dsl # include the `experiment` method

  def welcome
    @user = params[:user]

    ex = experiment(:project_suggestions, actor: @user) do |e|
      e.use { 'welcome' }
      e.try { 'welcome_with_project_suggestions' }
    end

    mail(to: @user.email, subject: 'Welcome!', template: ex.run)
  end
end

Context migrations

There are times when we need to change context while an experiment is running. We make this possible by passing the migration data to the experiment.

Take for instance, that you might be using version: 1 in your context currently. To migrate this to version: 2, provide the portion of the context you wish to change using a migrated_with option.

In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant.

# Migrate just the `:version` portion of the previous context, `{ actor: project, version: 1 }`:
experiment(:my_experiment, actor: project, version: 2, migrated_with: { version: 1 })

You can add or remove context by providing a migrated_from option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.

If you wanted to introduce a version to your context, provide the full previous context.

# Migrate the full context from `{ actor: project }` to `{ actor: project, version: 1 }`:
experiment(:my_experiment, actor: project, version: 1, migrated_from: { actor: project })

This can impact an experience if you:

1. haven't implemented the concept of migrations in your variant resolver
2. haven't enabled a reasonable caching mechanism

When there isn't an actor (cookie fallback)

When there isn't an identifying key in the context (this is actor by default), we fall back to cookies to provide a consistent experience for the client viewing them.

Once we assign a certain variant to a context, we need to always provide the same experience. We achieve this by setting a cookie for the experiment in question, but only when needed.

This cookie is a temporary, randomized uuid and isn't associated with a user. When we can finally provide an actor, the context is auto migrated from the cookie to that actor.

To read and write cookies, we provide the request from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers.

You'll need to provide the request as an option to the experiment if it's outside of the controller and views.

experiment(:my_experiment, actor: user, request: request)

The cookie isn't set if the actor key isn't present at all in the context. Meaning that when no actor key is provided, the cookie will not be set.

# actor is not present, so no cookie is set
experiment(:my_experiment, project: project)

# actor is present and is nil, so the cookie is set and used
experiment(:my_experiment, actor: nil, project: project)

# actor is present and set to a value, so no cookie is set
experiment(:my_experiment, actor: user, project: project)

For edge cases, you can pass the cookie through by assigning it yourself -- e.g. actor: request.cookie_jar.signed['my_experiment_actor']. The cookie name is the full experiment name (including any configured prefix) with _actor appended -- e.g. gitlab_notification_toggle_actor for the :notification_toggle experiment key with a configured prefix of gitlab.

Configuration

This gem needs to be configured before being used in a meaningful way.

The default configuration will always render the control, so it's important to configure your own logic for resolving variants.

Yes, the most important aspect of the gem -- that of determining which variant to render and when -- is up to you. Consider using Unleash or Flipper for this.

Gitlab::Experiment.configure do |config|
  # The block here is evaluated within the scope of the experiment instance,
  # which is why we are able to access things like name and context.
  config.variant_resolver = lambda do |requested_variant|
    # Return the requested variant if a specific one has been provided in code.
    return requested_variant unless requested_variant.nil?

    # Ask Unleash to determine the variant, given the context we've built,
    # using the control as the fallback.
    fallback = Unleash::Variant.new(name: 'control', enabled: true)
    UNLEASH.get_variant(name, context.value, fallback)
  end
end

More examples for configuration are available in the provided rails initializer.

Client layer / JavaScript

This library doesn't attempt to provide any logic for the client layer.

Instead it allows you to do this yourself in configuration. Using Gon to publish your experiment information to the client layer is pretty simple.

Gitlab::Experiment.configure do |config|
  config.publishing_behavior = lambda do |_result|
    # Push the experiment knowledge into the front end. The signature contains
    # the context key, and the variant that has been determined.
    Gon.push({ experiment: { name => signature } }, true)
  end
end

In the client you can now access window.gon.experiment.notificationToggle.

Caching

Caching can be enabled in configuration, and is implemented towards the Rails.cache / ActiveSupport::Cache::Store interface. When you enable caching, any variant resolution will be cached. Migrating the cache through context migrations is handled automatically, and this helps ensure an experiment experience remains consistent.

It's important to understand that using caching can drastically change or override your rollout strategy logic.

Gitlab::Experiment.configure do |config|
  config.cache = Rails.cache
end

Tracking, anonymity and GDPR

We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a. the context key).

We generate this key from the context passed to the experiment. This allows creating funnels without exposing any user information.

This library attempts to be non-user-centric, in that a context can contain things like a user or a project.

If you only include a user, that user would get the same experience across every project they view. If you only include the project, every user who views that project would get the same experience.

Each of these approaches could be desirable given the objectives of your experiment.

Development

After checking out the repo, run bundle install to install dependencies. Then, run bundle exec rspec to run the tests. You can also run bundle exec pry for an interactive prompt that will allow you to experiment.

Contributing

Bug reports and merge requests are welcome on GitLab at https://gitlab.com/gitlab-org/gitlab-experiment. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

Release Process

Please refer to the Release Process.

License

The gem is available as open source under the terms of the MIT License.

Code of Conduct

Everyone interacting in the Gitlab::Experiment project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.

Make code not war