Build Status Coverage Status

RSpock

RSpock is a testing and specification framework built on top of Minitest. It intends to give back productivity to its users with its incredibly simple yet highly expressive specification language.

Note: RSpock is heavily inspired by Spock for the Groovy programming language.

Goals

  • High readability, expressiveness, maintainability and productivity: Take back your very precious developer time!
  • Encourage code reuse through expressive data-driven tests

Features

  • BDD-style code blocks: Given, When, Then, Expect, Cleanup, Where
  • Data-driven testing with incredibly expressive table-based Where blocks
  • Expressive assertions: Use familiar comparison operators == and != for assertions!
  • Interaction-based testing, i.e. 1 * object.receive("message") in Then blocks
  • (Planned) BDD-style custom reporter that outputs information from Code Blocks
  • (Planned) Capture all Then block violations

Installation

Add this line to your application's Gemfile:

gem 'rspock'

And then execute:

$ bundle

Or install it yourself as:

$ gem install rspock

Add this to the very beginning of your script or application to install the ASTTransform hook:

require 'ast_transform'
ASTTransform.install

If you're in a non-Rails project, add this to your Rakefile to enable the Truth Table Generator Rake task:

require 'rspock'

spec = Gem::Specification.find_by_name 'rspock'
rakefile = "#{spec.gem_dir}/lib/Rakefile"
load rakefile

Rails

If you are using Rails, it is necessary to add a filter to Rails.backtrace_cleaner for source mapping to work, so that you get proper line numbers in Minitest backtraces. For your convenience, we've built a Rails Generator just for that:

$ rails g rspock:install

Note: If you are not using Rails, you don't have anything to do, as RSpock includes a Minitest plugin that will set its own backtrace filter.

Usage

Getting started using RSpock is extremely easy!

require 'test_helper'

transform!(RSpock::AST::Transformation)
class MyTest < Minitest::Test
  # Feature Methods go here
end

Note: transform! is an annotation added by the ast_transform module that allows executing AST transformations on the annotated code. See here for more info.

Example With Feature Method and Code Blocks

require 'test_helper'

transform!(RSpock::AST::Transformation)
class MyTest < Minitest::Test
  test "adding 1 and 2 results in 3" do
    When "Adding 1 and 2"
    actual = 1 + 2

    Then "We get the expected result 3"
    actual == 3
  end
end

Feature Methods

test "adding 1 and 2 results in 3" do
  # Code Blocks go here
end

A feature method consists of four main conceptual phases:

  • Setup
  • Provide a stimulus to the system under test
  • Describe the response expected from the system
  • Cleanup

The first and last steps are optional, however the stimulus and response phases are always present.

Code Blocks

Blocks Phases
Given Setup
When Stimulus
Then Response
Expect Stimulus + Response
Cleanup Cleanup
Where Repeat

RSpock has support for each conceptual phase of a feature method. As such, feature methods are structured into Code Blocks, each representing a phase.

See below diagram for how you may arrange code blocks. Any directed path from Start to End is valid.

Given Block

Given "An empty Cart and a Product"
cart = Cart.new
product = Product.new

The Given block is where you do any special setup for the Feature Method. It otherwise doesn't have any special semantics.

When Block

When "Adding a product"
cart.add_product(Product.new)

The When block describes the stimulus to be applied to the system under test. It is always followed by a Then block.

Then Block

Then "The product is added to the cart"
cart.products.size == 1
cart.products.first == product

The Then block describes the response from the stimulus. Any comparison operators used in the Then block (== or !=) is transformed to assert_equal / refute_equal under the hood. By convention, the LHS operand is considered the actual value, while the RHS operand is considered the expected value.

Expect Block

The Expect block is useful when expressing the stimulus and the response in one statement is more natural. For example, let's compare two equivalent ways of describing some behaviour:

When + Then
When "Calling #abs on a negative number"
actual = -2.abs

Then "Value is positive"
actual == 2
Using Expect
Expect "absolute of -2 is 2"
-2.abs == 2

A good rule of thumb is using When + Then blocks to describe methods with side-effects and Expect blocks to describe purely functional methods.

Cleanup Block

Given "Open the file"
file = File.new("/invalid/file/path") # raises

# other blocks...

Cleanup
file&.close # Use safe navigation operator, since +file+ is nil if an error occurred.

The Cleanup block is where you free any resources used by a Feature Method. It runs even if a previous part of the Feature Method produced an exception. This means that Cleanup blocks must be coded defensively so as to not raise NoMethodError. A good way to do this in Ruby is demonstrated above by using the &. safe navigation operator.

This is also useful to ensure a shared resource is cleaned in between Feature Methods / test runs so as to not leak test state.

Where Block

Where blocks have very special semantics in RSpock. They take the form of a data table, for readability.

Take a look at the following Feature Method for an example of how to use it:

test "Adding #{a} and #{b} results in #{c}" do
  When "Adding two numbers"
  actual = a + b

  Then "We get the expected result"
  actual == c

  Where
  a  | b  | c
  -1 | -1 | -2
  -1 | 0  | -1
  -1 | 1  | 0
  0  | -1 | -1
  0  | 0  | 0
  0  | 1  | 1
  1  | -1 | 0
  1  | 0  | 1
  1  | 1  | 2
end

The first row in the Where block is considered the Header. The names of columns will expose a local variable of the same name in the scope of the Feature Method. The header column names have the same constraints as method names in Ruby. Each other row defines one test case that will be generated, binding each column's data to the appropriate variable.

This effectively creates one version of the Feature Method for each data row. Note how we've listed test cases as if this was a truth table, ordering them by boolean increment. This makes it very easy to ensure all cases have been covered.

Note: Although the Where block is declared last, it is evaluated first. This means that it cannot access local variables previously defined in the test method. It is evaluated in Class scope, so it is possible to use generators or methods for column values, provided they are class methods, not instance methods.

Tip: By convention, the expected_result or output column should be the rightmost column.

Test Name Interpolation

You might have noticed above that the test name contains string interpolations, that's one of the features of RSpock! You can interpolate test names and use Where block header variables to parameterize the test name using the test data.

Truth Table

Formatting Where Blocks as a truth table is the recommended way to organize your Where Blocks in RSpock. It makes test cases more maintainable since it creates an order between the different test cases and makes it easier to spot missing or duplicated test cases. It also makes it easier to add a column:

  • Add your new column to the left with the first possible value
  • Copy all current Where Block's data lines
  • For each other possible value for the column you're adding, paste the copied lines
  • Fill the new column by iterating over its possible values and you're done!
Truth Table Generator

Are you a lazy programmer? Good, so are we!

If you end up in a situation where the code under test has many different inputs / states that make it hard to manually generate a Truth Table, or if you simply want to make sure you don't omit test cases, we got you covered! You can use our truth table generator to generate all the different test cases automatically!

Run the following command, listing columns in the order you want them to appear, and their possible values in the order you want them to be iterated on:

$ rake rspock:truth_table -- a=-1,0,1 b=-1,0,1 expected_result="'?'"

The above command outputs the following formatted table:

a  | b  | expected_result
-1 | -1 | '?'
-1 | 0  | '?'
-1 | 1  | '?'
0  | -1 | '?'
0  | 0  | '?'
0  | 1  | '?'
1  | -1 | '?'
1  | 0  | '?'
1  | 1  | '?'

Tip: When generating a truth table, we recommend to include the expected_result column with only one possible value, which you can then fill manually. Unless of course all your test cases have the same result, in which case you don't need said column.

Escaping Delimiter

You may escape the , delimiter, which can be useful when i.e. you're calling a method with multiple arguments to generate or build the data for a certain column.

$ rake rspock:truth_table -- a=0,1 b="generator(1\, 2)","generator(3\, 4)" expected_result="'?'"

The above command outputs the following formatted table:

a | b               | expected_result
0 | generator(1, 2) | '?'
0 | generator(3, 4) | '?'
1 | generator(1, 2) | '?'
1 | generator(3, 4) | '?'

Mocking with Interactions

Interaction-based testing is a testing practice which focuses on the messages sent to objects, rather than their state. It explores how the object(s) under specification interact, through method calls, with their collaborators.

RSpock allows a natural way of expressing these interactions, using the expected method calls as code directly. An example is worth a thousand words here:

test "#publish sends a message to all subscribers" do
  Given
  subscriber1 = Subscriber.new
  subscriber2 = Subscriber.new
  publisher = Publisher.new(subscriber1, subscriber2)

  When
  publisher.publish("message")

  Then
  1 * subscriber.receive("message")
  1 * subscriber2.receive("message")
end

The above Then block contains 2 interactions, each of which has 4 parts: the cardinality, the receiver, the message and its arguments.

1 * receiver.message('hello')
|   |          |       |
|   |          |       argument(s)
|   |          message
|   receiver
cardinality

Note: Interactions are supported in the Then block only.

Cardinality

The cardinality of an interaction describes how often a method is expected to be called. It can be a fixed number of times, or a range.

1 * receiver.message('hello')       # exactly once
0 * receiver.message('hello')       # must never be called
(1..3) * receiver.message('hello')  # between one and three times (inclusive)
(1...4) * receiver.message('hello') # between one and four times (exclusive) => same as (1..3) above
(1.._) * receiver.message('hello')  # at least once
(_..3) * receiver.message('hello')  # at most three times
_ * receiver.message('hello')       # any number of calls, including zero

Receiver

The receiver describes which object is expected to receive the method call.

1 * subscriber.receive('hello') # a call to 'subscriber'

Message

The message of an interaction describes which method is expected to be called on the receiver.

1 * subscriber.receive('hello') # a method named 'receive'

Arguments

The arguments of an interaction describe which arguments of the method call are expected.

1 * subscriber.receive('hello') # an argument that is equal to the String 'hello'

Debugging

Pry

Let's be honest, at some point you will need to debug your tests. Because RSpock requires transforming the AST, the executed code is slightly different from the source code that you wrote. Although we have plans to add Source Mapping support in Pry so that you can see the exact source code you wrote, this is not currently available. This means that code shown in the Pry console will be slightly different. We think the value of using RSpock greatly outweighs this current limitation. We encourage you to try debugging a test to see what the transformed code looks like, or look through tmp/rspock for the transformed files.

Backtraces

RSpock supports Source Mapping so that backtraces for the executed code point to the correct line numbers in your source code, and so that the correct files are referenced. This is achieved through wrapping the executed code in rescue blocks, processing the backtraces (source mapping) and re-raising the error.

Tips and Tricks

test_index and line_number

The generated test name for each test case will contain the test index and the line number, corresponding to the Where Block data row for that case, which is available in the test scope as _test_index_ and _line_number_ respectively. This can be leveraged to conditionally break on certain test cases, so that you can have a more granular debugging session.

test "Adding #{a} and #{b} results in #{c}" do
  When "Adding two numbers"
  actual = a + b

  Then "We get the expected result"
  # Breaks on the first test case
  binding.pry if _test_index_ == 0
  # Breaks on the second test case
  binding.pry if _line_number_ == 15
  actual == c

  Where
  a | b | c
  1 | 2 | 3
  4 | 5 | 9 # Line 15
end

A few notes:

  • Comparison with _test_index_ and _line_number_ is not transformed to assertions in Then and Expect Code Blocks
  • _test_index_ is zero-based, meaning the index of the first test case is 0

line_number

The Line number is extremely useful for figuring out exactly which test case failed in your Where Block, especially if you have many rows in your Where Block data table.

More info

RSpock Syntax Pre-Processor

transform!(RSpock::AST::Transformation)
class MyTest < Minitest::Test
  # ...
end

RSpock, although having valid Ruby syntax, has different semantics in certain contexts, so that we can offer a more expressive and readable syntax. As such, we perform AST transformations on the code to produce semantically valid-in-context Ruby code.

This is achieved by the transform! "method call" above. This doesn't actually call anything, it instead is an annotation exposed by the ast_transform module that picks up whatever AST transformations are passed as arguments and runs them before compilation into Ruby instructions. The transformations required by RSpock are encapsulated in RSpock::AST::Transformation.

Using RSpock Alongside Regular Minitest Tests

Although we strongly encourage being consistent in your code bases and using only RSpock Tests in the same test file, we understand that there might be a transition period until all tests in a file can be migrated to using RSpock.

To that effect, we support disabling the strict mode on the transform! annotation:

transform!(RSpock::AST::Transformation.new(strict: false))
class MyTest < Minitest::Test
  test "non-RSpock tests work" do
    assert_equal true, true
  end

  test "RSpock tests also work" do
    Expect
    1 + 2 == 3
  end
end

Development

After checking out the repo, run bin/setup to install dependencies. Then, run bundle exec rake test to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/rspockframework/rspock. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

The gem is available as open source under the terms of the MIT License.

Code of Conduct

Everyone interacting in the RSpock project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.