rspec-trace-formatter
An RSpec formatter for constructing trace data from your specs. This library is inspired by go-test-trace.
Why Would I Use This?
Collecting data from your RSpec tests may be useful to you for a number of reasons:
- You'd like to see statistical trends in test runtimes, or in your CI/CD pipeline as a whole
- You'd like a dataset containing pass/fail statuses for all tests to help hunt down flakes
- Other things that I can't think of
Traces aren't the only choice for collecting this data, but they are a reasonable one. With concepts like test files and example groups, test execution naturally maps onto a trace tree. The flexibility of tools like OpenTelemetry when it comes to including arbitrary key-value attribute pairings is useful when instrumenting a library like RSpec because we can preserve as much context about the tests as we like. And this test data isn't likely to be valuable long-term, so the standard retention periods for traces are likely to be acceptable.
What's In The Box?
There are three main parts to this library.
RSpec::Trace::Formatter
RSpec::Trace::Formatter
is an RSpec formatter that emits events containing data that's relevant for constructing traces.
This formatter doesn't create traces -- it only outputs JSON events.
This formatter emits events for the most significant lifecycle events in an RSpec suite: the start of the suite, the start/end of each example and example group, and the end of the suite. Because all events are timestamped, you can expect accurate timing data. It also collects data about the names of the examples and example groups that are run, as well as useful facts like file locations, pass/fail status, &etc.
The event format is designed to be redundant when providing facts about examples and example groups, so as to be less prescriptive about how you consume them. This may not be the best decision, but it seemed the right way.
rspec-trace-consumer
rspec-trace-consumer
is a script that reads events created by RSpec::Trace::Formatter
from standard input and emits trace data to an OpenTelemetry collector.
The OpenTelemetry SDK can be configured using the standard OTEL_*
environment variables.
If not set by the OTEL_SERVICE_NAME
environment variable, the service name will be set to rspec
.
The name of the root span defaults to "rspec", but you can change that as well with the RSPEC_TRACE_FORMATTER_ROOT_SPAN_NAME
environment variable.
This script uses the AlwaysOn
sampler to ensure that no data is ever discarded.
RSpec::Trace::OpenTelemetryFormatter
RSpec::Trace::OpenTelemetryFormatter
is a separate RSpec formatter that combines the two pieces above to collect trace events from RSpec tests and send them to an OpenTelemetry collector.
Because this uses the (very nice!) subprocess library, it only works on Ruby platforms where fork
is supported.
If you're running in an environment where this isn't supported (e.g. JRuby) you won't be able to use this.
However, the rest of this library is expected to work for you, and specifying an --out
target for RSpec::Trace::Formatter
may make this easier.
How Do I Use It?
You can install this gem by adding the rspec-trace-formatter
(along with the necessary OpenTelemetry dependencies, if they aren't already included) to your Gemfile
and running bundle install
.
For example:
group :test do
gem "rspec-trace-formatter"
gem "opentelemetry-api", "~> 1.0"
gem "opentelemetry-exporter-otlp", "~> 0.20.0"
end
This library should be used like any other RSpec formatter, with the assistance of any environment variables that you need to control the OpenTelemetry data.
Example of using the RSpec::Trace::OpenTelemetryFormatter
with representative environment variables set:
$ OTEL_TRACES_EXPORTER=console bundle exec rspec --format RSpec::Trace::OpenTelemetryFormatter
Example of running the RSpec::Trace::Formatter
by itself and sending the output to rspec-trace-consumer
separately (in a way that you can surely improve upon):
$ OTEL_TRACES_EXPORTER=console bundle exec rspec --format RSpec::Trace::Formatter --out /tmp/trace-events.jsonl
# Piping the input in
$ rspec-trace-consumer < /tmp/trace-events.jsonl
# Passing a filename as an argument
$ rspec-trace-consumer /tmp/trace-events.jsonl
If the TRACEPARENT
environment variable is set in either of these cases, it will be interpreted as a W3C Trace Context Traceparent Header value.
This will allow you to include the span events generated by this library in a larger distributed trace.
How Do I Contribute?
Very carefully, I hope.
One notable fact is that we use snapshot testing for the class underpinning rspec-trace-consumer
.
To keep this reliable, I've defined a custom OpenTelemetry span exporter that includes meaningful-enough data to test with and no execution-specific fields.
Useful rake
commands
rake build
: Build the gemrake install
: Builds and installs the gemrake regenerate_examples
: Rebuilds fixtures for snapshot testsrake test
: Runs the automated tests (written with RSpec, of course)rake update_snapshots
: Updates the test snapshots
Containers
Configuration for a dev container is provided for convenience.
The main practical benefit of developing in the container is to be able to regenerate the snapshots for the Consumer
tests with consistent and non-identifying file paths for the stack traces.
License
MIT