Hatchet
What
The Hatchet is a an integration testing library for developing Heroku buildpacks.
Install
First run:
$ bundle install
This library uses the heroku-api gem, you will need to make your API key available to the system.
You can get your token by running:
$ heroku auth:token
alskdfju108f09uvngu172019
We need to export this token into our environment open up your .bashrc
export HEROKU_API_KEY="alskdfju108f09uvngu172019"
Then source the file. If you don't want to set your api key system wide, it will be pulled automatically via shelling out, but this is slower.
Run the Tests
$ bundle exec rake test
Writing Tests
Hatchet is meant for running integration tests, which means we actually have to deploy a real live honest to goodness app on Heroku and see how it behaves.
First you'll need a repo to an app you know works on Heroku, add it to the proper folder in repos. Such as repos/rails3/
. Once you've done that make a corresponding test file in the test
dir such as test/repos/rails3
. I've already got a project called "codetriage" and the test is "triage_test.rb".
Now that you have an app, we'll need to create a heroku instance, and deploy our code to that instance you can do that with this code:
Hatchet::App.new("repos/rails3/codetriage").deploy do |app|
##
end
The first argument to the app is the directory where you can find the code. Once your test is done, the app will automatically be destroyed.
Now that you've deployed your app you'll want to make sure that the deploy worked correctly. Since we're using test/unit
you can use regular assertions such as assert
and refute
. Since we're yielding to an app
variable we can check the deployed?
status of the app:
Hatchet::App.new("repos/rails3/codetriage").deploy do |app|
assert app.deployed?
end
The primary purpose of the buildpack is configuring and deploying apps, so if it deployed chances are the buildpack is working correctly, but sometimes you may want more information. You can run arbitrary commands such as heroku bash
and then check for the existence of a file.
Hatchet::App.new("repos/rails3/codetriage").deploy do |app|
app.run("bash") do |cmd|
assert cmd.run("ls public/assets").include?("application.css")
end
end
Anything you put in cmd.run
at this point will be executed from with in the app that you are in.
cmd.run("cat")
cmd.run("cd")
cmd.run("cd .. ; ls | grep foo")
It behaves exactly as if you were in a remote shell. If you really wanted you could even run the tests:
cmd.run("rake test")
But since cmd.run doesn't return the exit status now, that wouldn't be so useful (also there is a default timeout to all commands). If you want you can configure the timeout by passing in a second parameter
cmd.run("rake test", 180.seconds)
Running One off Commands
If you only want to run one command you can call app.run
without
passing a block
Hatchet::AnvilApp.new("/codetriage").deploy do |app|
assert_match "1.9.3", app.run("ruby -v")
end
Testing A Different Buildpack
You can specify buildpack to deploy with like so:
Hatchet::App.new("repos/rails3/codetriage", buildpack: "https://github.com/schneems/heroku-buildpack-ruby.git").deploy do |app|
Hatchet Config
Hatchet is designed to test buildpacks, and requires full repositories
to deploy to Heroku. Web application repos, especially Rails repos, aren't known for
being small, if you're testing a custom buildpack and have
BUILDPACK_URL
set in your app config, it needs to be cloned each time
you deploy your app. If you've git add
-ed a bunch of repos then this
clone would be pretty slow, we're not going to do this. Do not commit
your repos to git.
Instead we will keep a structured file called
inventively hatchet.json
at the root of your project. This file will
describe the structure of your repos, have the name of the repo, and a
git url. We will use it to sync remote git repos with your local
project. It might look something like this
{
"hatchet": {},
"rails3": ["[email protected]:codetriage/codetriage.git"],
"rails2": ["[email protected]:heroku/rails2blog.git"]
}
the 'hatchet' object accessor is reserved for hatchet settings.
. To copy each repo in your hatchet.json
run the command:
$ hatchet install
The above hatchet.json
will produce a directory structure like this:
repos/
rails3/
codetriage/
#...
rails2/
rails2blog/
# ...
While you are running your tests if you reference a repo that isn't synced locally Hatchet will raise an error. Since you're using a standard file for your repos, you can now reference the name of the git repo, provided you don't have conflicting names:
Hatchet::App.new("codetriage").deploy do |app|
If you do have conflicting names, use full paths.
A word of warning on including rails/ruby repos inside of your test
directory, if you're using a runner that looks for patterns such as
*_test.rb
to run your hatchet tests, it may incorrectly think you want
to run the tests inside of the rails repositories. To get rid of this
problem move your repos direcory out of test/
or be more specific
with your tests such as moving them to a test/hatchet
directory and
changing your pattern if you are using Rake::TestTask
it might look like this:
t.pattern = 'test/hatchet/**/*_test.rb'
A note on external repos: since you're basing tests on these repos, it is in your best interest to not change them or your tests may spontaneously fail. In the future we may create a hatchet.lockfile or something to declare the commit
Hatchet CLI
Hatchet has a CLI for installing and maintaining external repos you're using to test against. If you have Hatchet installed as a gem run
$ hatchet --help
For more info on commands. If you're using the source code you can run the command by going to the source code directory and running:
$ ./bin/hatchet --help
The Future
Speed
Efforts may be spent optimizing / parallelizing the process, almost all of the time of the test is spent waiting for IO, so hopefully we should be able to parallelize many tests / deploys at the same time. The hardest part of this (i believe) would be splitting out the different runs into different log streams so that the output wouldn't be completely useless.
Right now running 1 deploy test takes about 3 min on my machine.
Git Based Deploys
It would be great to allow hatchet to deploy apps off of git url, however if we do that we could open ourselves up to false negatives, if we are pointing at an external repo that gets broken.
Features?
What else do we want to test? Config vars, addons, etc. Let's write some tests.