capybara-lockstep ![Tests](https://github.com/makandra/capybara-lockstep/actions/workflows/test.yml/badge.svg)
This Ruby gem synchronizes Capybara commands with client-side JavaScript and AJAX requests. This greatly improves the stability of an end-to-end ("E2E") test suite, even if that suite has timing issues.
The next section explains why your test suite is flaky and how capybara-lockstep can help.\ If you don't care you may skip to installation instructions.
Why are tests flaky?
A naively written E2E test will have race conditions between the test script and the controlled browser. How often these timing issues will cause your tests to fail depends on luck and your machine's performance. You may not see these issues for years until a colleague runs your suite on their new laptop.
Here is a typical example for a test that will fail with unlucky timing:
scenario 'User sends a tweet' do
visit '/'
click_link 'New tweet' # opens form in a modal dialog
fill_in 'text', with: 'My first tweet'
'Send tweet'
visit '/timeline'
expect(page).to have_css('.tweet', text: 'My first tweet')
end
This test has four timing issues that may cause it to fail:
- We click on the New tweet button, but the JS event handler to open the tweet form hasn't been registered yet.
- We start filling in the form, but it hasn't been loaded yet.
- After sending the tweet we immediately navigate away, killing the form submission request that is still in flight. Hence the tweet will never appear in the next step.
- We look for the new tweet, but the timeline hasn't been loaded yet.
Capybara will retry individual commands or expectations when they fail.\ However, only issues 2 and 4 can be healed by retrying.
While it is possible to remove most of the timing issues above, it requires skill and discipline.\ capybara-lockstep fixes issues 1, 2, 3 and 4 without any changes to the test code.
This is a JavaScript problem
The timing issues above will only manifest in an app where links, forms and buttons are handled by JavaScript.
When all you have is standard HTML links and forms, stock Capybara will not see timing issues:
- After a
visit()
Capybara/WebDriver will wait until the page is completely loaded - When following a link Capybara/WebDriver will wait until the link destination is completely loaded
- When submitting a form Capybara/WebDriver will wait until the response is completely loaded
However, when JavaScript handles a link click, you get zero guarantees.\ Capybara/WebDriver will not wait for AJAX requests or any other async work.
How capybara-lockstep helps
capybara-lockstep waits until the browser is idle before moving on to the next Capybara command. This greatly relieves the pressure on Capybara's retry logic.
capybara-lockstep synchronizes when one of the following occurs:
- Capybara looks up an element
- Capybara simulates a user interaction (clicking, typing, etc.)
- Capybara visits a new URL
- Capybara executes JavaScript
When capybara-lockstep synchronizes it will:
- wait for all document resources to load (images, CSS, fonts, frames).
- wait for client-side JavaScript to render or hydrate DOM elements.
- wait for any pending AJAX requests to finish and their callbacks to be called.
- wait for dynamically inserted
<script>
s to load (e.g. from dynamic imports or Analytics snippets). - wait for dynamically inserted
<img>
or<iframe>
elements to load (ignoring lazy-loaded elements).
In summary, Capybara can no longer observe or interact with the page while HTTP requests are in flight. This covers most async work that causes flaky tests.
Limitations
Async work not synchronized by capybara-lockstep includes:
- Animations
- Websocket connections
- Service workers
- Work scheduled via
setTimeout()
orsetInterval()
. <audio>
and<video>
elements
You can configure capybara-lockstep to wait for additional async work.
Installation
Prerequisites
Check if your application satisfies all requirements for capybara-lockstep:
- Capybara 2.0 or higher.
- Your Capybara driver must use selenium-webdriver 3.0 or higher. capybara-lockstep deactivates itself for any other driver. There is a fork with support for capybara-playwright-driver.
- This gem was only tested with a Selenium-controlled Chrome browser. Chrome in headless mode is recommended, but not required.
- This gem was only tested with Rails, but there's no Rails dependency.
Installing the Ruby gem
Assuming that you're using Rails, add this to your application's Gemfile
:
group :test do
gem 'capybara-lockstep'
end
And then execute:
$ bundle install
If you're not using Rails you should also require 'capybara-lockstep'
in your spec_helper.rb
(RSpec), test_helper.rb
(Minitest) or env.rb
(Cucumber).
Including the JavaScript snippet (required)
capybara-lockstep requires a JavaScript snippet to be embedded by the application under test. If that snippet is missing on a screen, capybara-lockstep will not be able to synchronize with the browser. In that case the test will continue without synchronization.
If you're using Rails you can use the capybara_lockstep
helper to insert the snippet into your application layouts:
<%= capybara_lockstep if defined?(Capybara::Lockstep) %>
Ideally the snippet should be included in the <head>
before any other <script>
tags.
If you're not using Rails you can include Capybara::Lockstep::Helper
and access the JavaScript code with capybara_lockstep_js
.
If you have a strict Content Security Policy the capybara_lockstep
Rails helper will insert a CSP nonce by default. You can also pass an explicit nonce string using the :nonce
option.
Including the middleware (optional)
This gem provides Rack middleware to block Capybara while your Rails (or Rack) backend is busy.
Using the middleware is optional, as the JavaScript snippet already waits for asynchronous work on the client. However, using the middleware covers some additional edge cases. For example, the middleware detects requests that were aborted on the frontend, but are still being processed by the backend.
To include the middleware in a Rails application, add the following line to config/environments/test.rb
:
config.middleware.insert_before 0, Capybara::Lockstep::Middleware
In a non-Rails application you should include the middleware as high up in your middleware stack as possible:
use Capybara::Lockstep::Middleware
# Other middleware here
Configuring Selenium WebDriver (recommended)
By default, webdrivers will automatically dismiss any user prompts (like alerts) when trying to perform an action. While capybara-lockstep carefully detects alerts before synchronizing, and will skip interaction with the browser to avoid accidentally dismissing alerts, it can not synchronize around some rare race conditions.
We recommend you configure your webdriver to not automatically dismiss user prompts by setting the "unhandled prompt behavior" capability to ignore
. Using "ignore", errors are raised like with the default behavior, but user prompts are kept open.
For example, the Chrome driver can be configured like this:
Capybara.register_driver(:selenium) do |app|
= Selenium::WebDriver::Chrome::Options.new(
unhandled_prompt_behavior: 'ignore',
# ...
)
Capybara::Selenium::Driver.new(app, browser: :chrome, options: )
end
Verify successful integration
capybara-lockstep will automatically patch Capybara to wait for the browser after every command.
Run your test suite to see if integration was successful and whether stability improves. During validation we recommend to activate the debugging log before your test:
Capybara::Lockstep.debug = true
You should see messages like this in your console:
[capybara-lockstep] Synchronizing
[capybara-lockstep] Finished waiting for JavaScript
[capybara-lockstep] Synchronized successfully
Note that you may see some failures from tests with wrong assertions, which previously passed due to lucky timing.
Signaling asynchronous work
By default capybara-lockstep waits until resources have loaded, AJAX requests have finished and their callbacks have been called. There are also some limitations.
You can configure capybara-lockstep to wait for other async work.
On the frontend
Let's say we have an animation that fades in a new element over 2 seconds. The following will block Capybara while the animation is running:
async function fadeIn(element) {
CapybaraLockstep?.startWork('Animation')
startAnimation(element, 'fade-in')
await waitForAnimationEnd(element)
CapybaraLockstep?.stopWork('Animation')
}
The string argument is used for logging (when logging is enabled). It does not need to be unique per job. In this case you should see messages like this in your browser's JavaScript console:
[capybara-lockstep] Started work: Animation [1 jobs]
[capybara-lockstep] Finished work: Animation [0 jobs]
You may omit the string argument, in which case nothing will be logged, but the work will still be tracked.
On the backend
You don't need to signal work within the regular request/response cycle, as this is detected automatically. You can however signal work that happens outside a request, e.g. in a background job or WebSocket handler.
The following will block Capybara while a Sidekiq job is running:
class HardJob
include Sidekiq::Job
def perform(name, count)
Capybara::Lockstep.start_work('HardJob') if defined?(Capybara::Lockstep)
# do something
ensure
Capybara::Lockstep.stop_work('StopWork') if defined?(Capybara::Lockstep)
end
end
Performance impact
capybara-lockstep may or may not impact the runtime of your test suite. It depends on your particular tests and how many flaky tests you're seeing in the first place.
While waiting for the browser to be idle does take a few milliseconds, Capybara no longer needs to retry failed commands. You will also save time from not needing to re-run failed tests.
In casual testing with large test suites I experienced a performance impact between +/- 10%.
Debugging log
You can enable extensive logging. This is useful to see whether capybara-lockstep has an effect on your tests, or to debug why synchronization is taking too long.
To enable the log, say this before or during a test:
Capybara::Lockstep.debug = true
You should now see messages like this on your standard output:
[capybara-lockstep] Synchronizing
[capybara-lockstep] Finished waiting for JavaScript
[capybara-lockstep] Synchronized successfully
You should also see messages like this in your browser's JavaScript console:
[capybara-lockstep] Started work: fetch /path [3 jobs]
[capybara-lockstep] Finished work: fetch /path [2 jobs]
Using a logger
You may also configure logging to an existing logger object:
Capybara::Lockstep.debug = Rails.logger
Logging in the browser only
To enable logging in the browser console (but not STDOUT), include the JavaScript snippet with { debug: true }
:
(debug: true)
Synchronization timeout
By default capybara-lockstep will wait Capybara.default_max_wait_time
seconds for the page initialize and for JavaScript and AJAX request to finish.
When synchronization times out, capybara-lockstep will log:
[capybara-lockstep] Could not synchronize within 3 seconds
You can configure a different timeout:
Capybara::Lockstep.timeout = 5 # seconds
By default Capybara will not raise an error after a timeout. You may occasionally get a slow server response, and Capybara will retry synchronization before the next interaction or visit
. This is often good enough.
If you want to be strict you may configure that an Capybara::Lockstep::Timeout
error is raised after a timeout:
Capybara::Lockstep.timeout_with = :error
To revert to defaults:
Capybara::Lockstep.timeout = nil
Capybara::Lockstep.timeout_with = nil
Manual synchronization
capybara-lockstep will automatically patch Capybara to wait for the browser after every command. This should be enough for most test suites.
For additional edge cases you may manually tell capybara-lockstep to wait. The following Ruby method will block until the browser is idle:
Capybara::Lockstep.synchronize
You may also synchronize from your client-side JavaScript. The following will run the given callback once the browser is idle:
CapybaraLockstep.synchronize(callback)
Disabling synchronization
Sometimes you want to disable browser synchronization, e.g. to observe a loading spinner during a long-running request.
To disable automatic synchronization:
begin
Capybara::Lockstep.mode = :manual
do_unsynchronized_work
ensure
Capybara::Lockstep.mode = :auto
end
You can also disable automatic synchronization for the duration of a block:
Capybara::Lockstep.with_mode(:manual) do
do_unsynchronized_work
end
In the :manual
mode you may still force synchronization by calling Capybara::Lockstep.synchronize
manually:
Capybara::Lockstep.with_mode(:manual) do
do_some_work
Capybara::Lockstep.synchronize
do_other_work
end
To completely disable synchronization, even when Capybara::Lockstep.synchronize
is called:
Capybara::Lockstep.mode = :off
Capybara::Lockstep.synchronize # will not synchronize
Handling legacy promises
Legacy promise implementations (like jQuery's $.Deferred
and AngularJS' $q
) work using tasks instead of microtasks. Their AJAX implementations (like $.ajax()
and $http
) use task-based promises to signal that a request is done.
This means there is a time window in which all AJAX requests have finished, but their callbacks have not yet run:
$http.get('/foo').then(function() {
// This callback runs one task after the response was received
})
It is theoretically possible that your test will observe the browser in that window, and expect content that has not been rendered yet. Affected code must call then()
on a task-based promise or use setTimeout()
to push work into the next task.
Any issues caused by this will usually be mitigated by Capybara's retry logic. If you think that this is an issue for your test suite, you can configure capybara-headless to wait additional tasks before it considers the browser to be idle:
Capybara::Lockstep.wait_tasks = 2 # default is 1
If you see longer chains of then()
or nested setTimeout()
calls in your code, you may need to configure a higher number of tasks to wait.
Waiting additional tasks will have a negative performance impact on your test suite.
Running code after synchronization
You can configure a proc to run after successful synchronization:
Capybara::Lockstep.after_synchronize do
puts "Synchronized!"
end
Contributing
Pull requests are welcome on GitHub at https://github.com/makandra/capybara-lockstep.
After checking out the repo, run bin/setup
to install dependencies.
Then, run rake spec
to run the tests.
You can also run bin/console
for an interactive prompt that will allow you to experiment.
Manually testing a change
To test an unrelased change with a test suite, we recommend to temporarily link the local repository from your test suites's Gemfile
:
gem 'capybara-lockstep', path: '../capybara-lockstep'
As an alternative you may also install this gem onto your local machine by running bundle exec rake install
.
Releasing a new version
- Update the version number in
version.rb
- Run
bundle exec rake release
, which will create a git tag for the version, push git commits and the created tag, and push the.gem
file to rubygems.org. - If RubyGems publishing seems to freeze, try entering your OTP code.
- Run
License
The gem is available as open source under the terms of the MIT License.
Credits
Henning Koch (@triskweline) from makandra.