TLDR - for people who don't have time for slow tests

Okay, you might need to sit down for this:

tl;dr, TLDR is a Ruby test framework that stops running your tests after 1.8 seconds.

We initially meant this as a joke while pairin', but in addition to being funny, it was also a pretty good idea. So we fleshed out tldr to be a full-featured, mostly Minitest-compatible, and dare-we-say pleasant test framework for Ruby.

The "big idea" here is TLDR is designed for users to run the tldr command repeatedly as they work—as opposed to only running the tests for whatever is being worked on. Even if the suite runs over the 1.8 second time limit. Because TLDR shuffles and runs in parallel and is guaranteed to take less than two seconds, you'll actually wind up running all of your tests quite often as you work, catching any problems much earlier than if you had waited until the end of the day to push your work and let a continuous integration server run the full suite.

Some stuff you might like:

  • A CLI that can specify tests by line number(s) (e.g. foo.rb:5 bar.rb:3:10) and by names or patterns (e.g. --name test_fail,test_error --name "/_\d/")
  • Everything is parallel by default, and seems pretty darn fast; TLDR also provides several escape hatches to sequester tests that aren't thread-safe
  • Surprisingly delightful color diff output when two things fail to equal one another, care of @mcmire's super_diff gem
  • By default, the CLI will prepend your most-recently-edited test file to the front of your suite so its tests will run first. The test you worked on most recently is the one you most likely want to ensure runs, so TLDR runs it first (see the --prepend option for how to control this behavior)
  • And, of course, our signature feature: your test suite will never grow into a glacially slow, soul-sucking albatross around your neck, because after 1.8 seconds, it stops running your tests, with a report on what it was able to run and where your slowest tests are

Some stuff you might not like:

  • The thought of switching Ruby test frameworks in 2023
  • That bit about your test suite exploding after 1.8 seconds

Install

Either gem install tldr or add it to your Gemfile:

gem "tldr"

Usage

Here's what a test looks like:

class MathTest < TLDR
  def test_adding
    assert_equal 1 + 1, 2
  end
end

A TLDR subclass defines its tests with instance methods that begin with test_. They can define setup and/or teardown methods which will run before and after each test, respectively.

If you place your tests in test/**/*_test.rb (and/or test/**/test_*.rb) files, the tldr executable will find them automatically. And if you define a test/helper.rb file, it will be loaded prior to your tests.

Running the CLI is pretty straightforward:

$ tldr

You can, of course, also just run a specific test file or glob:

$ tldr test/this/one/in/particular.rb

Or specify the line numbers of tests to run by appending them after a :

$ tldr test/fixture/line_number.rb:3:10

And filter which tests run by name or pattern with one or more --name or -n flags:

$ tldr --name FooTest#test_foo -n test_bar,test_baz -n /_qux/

(The above will translate to this array of name filters internally: ["FooTest#test_foo", "test_bar", "test_baz", "/_qux/"].)

Options

Here is the full list of CLI options:

$ tldr --help
Usage: tldr [options] some_tests/**/*.rb some/path.rb:13 ...
        --fail-fast                  Stop running tests as soon as one fails
    -s, --seed SEED                  Seed for randomization
        --[no-]parallel              Parallelize tests (Default: true)
    -n, --name PATTERN               One or more names or /patterns/ of tests to run (like: foo_test, /test_foo.*/, Foo#foo_test)
        --exclude-name PATTERN       One or more names or /patterns/ NOT to run
        --exclude-path PATH          One or more paths NOT to run (like: foo.rb, "test/bar/**", baz.rb:3)
        --helper PATH                One or more paths to a helper that is required before any tests (Default: "test/helper.rb")
        --no-helper                  Don't require any test helpers
        --prepend PATH               Prepend one or more paths to run before the rest (Default: most recently modified test)
        --no-prepend                 Don't prepend any tests before the rest of the suite
    -l, --load-path PATH             Add one or more paths to the $LOAD_PATH (Default: ["lib", "test"])
    -r, --reporter REPORTER          Set a custom reporter class (Default: "TLDR::Reporters::Default")
        --base-path PATH             Change the working directory for all relative paths (Default: current working directory)
        --no-dotfile                 Disable loading .tldr.yml dotfile
        --no-emoji                   Disable emoji in the output
    -v, --verbose                    Print stack traces for errors
        --print-interrupted-test-backtraces
                                     Print stack traces for interrupted tests
        --[no-]warnings              Print Ruby warnings (Default: true)
        --watch                      Run your tests continuously on file save (requires 'fswatch' to be installed)
        --yes-i-know                 Suppress TLDR report when suite runs over 1.8s
        --i-am-being-watched         [INTERNAL] Signals to tldr it is being invoked under --watch mode
        --comment COMMENT            [INTERNAL] No-op; used for multi-line execution instructions

After being parsed, all the CLI options are converted into a TLDR::Config object.

Setting defaults in .tldr.yml

The tldr CLI will look for a .tldr.yml file in your project root (your working directory or whatever --base-path you set), which can contain values for any properties on TLDR::Config (with the exception of --base-path itself).

Any values found in the dotfile will override TLDR's built-in values, but can still be specified by the tldr CLI or a TLDR::Config object passed to TLDR::Run.at_exit!.

Here's an example project that specifies a .tldr.yml file as well as some internal tests demonstrating its behavior.

Minitest compatibility

Tests you write with tldr are designed to be mostly-compatible with Minitest tests. Some notes:

  • setup and teardown hook methods should work as you expect. (We even threw in an around hook as a bonus!)
  • All of Minitest's assertions (e.g. assert, assert_equals) are provided, with these caveats:
    • To retain the expected, actual argument ordering, tldr defines assert_include?(element, container) instead of assert_includes(container, element)
    • If you want to maximize compatibility and mix in assert_includes and the deprecated assert_send, just include TLDR::Assertions::MinitestCompatibility into the TLDR base class or individual test classesJust set it

Running tests continuously with --watch

The tldr CLI includes a --watch option which will watch for changes in any of the configured load paths (["test", "lib"] by default) and then execute your tests each time a file is changed. To keep the output up-to-date and easy to scan, it will also clear your console before each run.

Note that this feature requires you have fswatch installed and on your PATH

Here's what that might look like:

tldr-watch

Running TLDR with Rake

TLDR ships with a very minimal rake task that simply shells out to the tldr CLI. If you want to run TLDR with Rake, you can configure the test run by setting flags on an env var named TLDR_OPTS or else in the .tldr.yml.

Here's an example Rakefile:

require "standard/rake"
require "tldr/rake"

task default: [:tldr, "standard:fix"]

You could then run the task with:

$ TLDR_OPTS="--no-parallel" bundle exec rake tldr

One reason you'd want to invoke TLDR with Rake is because you have multiple test suites that you want to be able to conveniently run separately (this talk discussed a few reasons why this can be useful).

To create a custom TLDR Rake test, just instantiate TLDR::Task like this:

require "tldr/rake"

TLDR::Task.new(name: :safe_tests, config: TLDR::Config.new(
  paths: FileList["safe/**/*_test.rb"],
  helper_paths: ["safe/helper.rb"],
  load_paths: ["lib", "safe"]
))

The above will create a second Rake task named safe_tests running a different set of tests than the default tldr task. Here's an example.

Running tests without the CLI

If you'd rather use TLDR by running Ruby files instead of the tldr CLI (similar to require "minitest/autorun"), here's how to do it!

Given a file test/some_test.rb:

require "tldr"
TLDR::Run.at_exit! TLDR::Config.new(no_emoji: true)

class SomeTest < TLDR
  def test_truth
    assert true
  end
end

You could run the test with:

$ ruby test/some_test.rb

To maximize control and to avoid running code accidentally (and unlike the tldr CLI), running at_exit! will not set default values to the paths, helper, load_paths, and prepend_paths config properties. You'll have to pass any values you want to set on a Config object and pass it to at_exit!.

To avoid running multiple suites accidentally, if TLDR::Run.at_exit! is encountered multiple times, only the first hook will be registered. If the tldr CLI is running and encounters a call to at_exit!, it will be ignored.

Setting up the load path

By default, the tldr CLI adds test and lib directories to the load path for you, but when running TLDR from a Ruby script, it doesn't set those up for you.

If you want to require code in test/ or lib/ without using require_relative, you'll need to add those directories to the load path. You can do this programmatically by prepending the path to $LOAD_PATH, like this:

$LOAD_PATH.unshift "test"

require "tldr"
TLDR::Run.at_exit! TLDR::Config.new(no_emoji: true)

require "helper"

Or by using Ruby's -I flag to include it:

$ ruby -Itest test/some_test.rb

Questions you might be asking

TLDR is very similar to Minitest in API, but different in enough ways that you probably have some questions.

Parallel-by-default is nice in theory but half my tests are failing. Wat?

Read this before you add --no-parallel because some tests are failing when you run tldr.

The vast majority of test suites in the wild are not parallelized and the vast majority of those will only parallelize by forking processes as opposed to using a thread pool. We wanted to encourage more people to save time (after all, you only get 1.8 seconds here) by making your test suite run as fast as it can, so your tests run in parallel threads by default.

If you're writing new code and tests with TLDR and dutifully running tldr constantly for fast feedback, odds are that this will help you catch thread safety issues early—this is a good thing, because it gives you a chance to address them before they're too hard to fix! But maybe you're porting an existing test suite to TLDR and running in parallel for the first time, or maybe you need to test something that simply can't be exercised in a thread-safe way. For those cases, TLDR's goal is to give you some tools to prevent you from giving up and adding --no-parallel to your entire test suite and slowing everything down for the sake of a few tests.

So, when you see a test that is failing when run in parallel with the rest of your suite, here is what we recommend doing, in priority order:

  1. Figure out a way to redesign the test (or the code under test) to be thread-safe. Modern versions of Ruby provide a number of tools to make this easier than it used to be, and it may be as simple as making an instance variable thread-local
  2. If the problem is that a subset of your tests depend on the same resource, try using TLDR.run_these_together! class to group the tests together. This will ensure that those tests run in the same thread in sequence (here's a simple example)
  3. For tests that affect process-wide resources like setting the system clock or changing the process's working directory (i.e. Dir.chdir), you can sequester them to run sequentially after all parallel tests in your suite have run with TLDR.dont_run_these_in_parallel!, which takes the same arguments as run_these_together! (example)
  4. Give up and make the whole suite --no-parallel. If you find that you need to resort to this, you might save some keystrokes by adding parallel: false in a .tldr.yml file

We have a couple other ideas of ways to incorporate non-thread-safe tests into your suite without slowing down the rest of your tests, so stay tuned!

How will I run all my tests in CI without the time bomb going off?

TLDR will run all your tests in CI without the time bomb going off. If tldr is run in a non-interactive shell and a CI environment variable is set (as it is on virtually every CI service), then the bomb will be defused.

What if I already have another tldr executable on my path?

There's a command-line utility named tldr that might conflict with this gem's binary in your PATH. If that's the case you could change your path, invoke bundle exec tldr, run with Rake, or use the tldt ("too long; didn't test") executable alias that ships with this gem.

Is there a plugin system?

There is not.

Currently, the only pluggable aspect of TLDR are reporters, which can be set with the --reporter command line option. It can be set to any fully-qualified class name that extends from TLDR::Reporters::Base.

I know my tests are over 1.8s, how do I suppress the huge output?

Plenty of test suites are over 1.8s and having TLDR repeatedly print out the huge summary at the end of each test run can be distracting and make it harder to spot test failures. If you know your test suite is too slow, you can simply add the --yes-i-know flag

What about mocking?

TLDR is laser-focused on running tests, so it doesn't provide a built-in mocking facility. Might we interest you in a refreshing mocktail, instead?

Contributing to TLDR

If you want to submit PRs on this repo, please know that the code style is Kirkland-style Ruby, where method definitions have parentheses omitted but parentheses are generally expected for method invocations.

Acknowledgements

Thanks to George Sheppard for freeing up the tldr gem name!