Resque recipe for EY Cloud

Resque is a Redis-backed Ruby library for creating background jobs, placing those jobs on multiple queues, and processing them later.

This recipe will setup Resque on a Solo instance environment or on named Utility instances in a cluster environment.

Installation

  1. Install the recipe
  2. Add a deploy hook to restart resque workers after the application restarts
  3. Name all utility instances resque.
# deploy/after_restart.rb
on_utilities(:resque) do
  node[:applications].each do |app_name, data|
    sudo 'echo "sleep 20 && monit -g resque_#{app_name} restart all" | at now'
  end
end

Simple Installation

To add this recipe to your collection of recipes, or as your first recipe, you can use the helpful ey-recipes command line tool:

cd myapp
gem install engineyard engineyard-recipes
ey-recipes init
ey-recipes clone git://github.com/engineyard/eycloud-recipe-resque.git -n resque
ey recipes upload --apply

If you want to have your recipes run during deploy (rather than the separate ey recipes upload --apply step):

ey-recipes init -d
ey-recipes clone git://github.com/engineyard/eycloud-recipe-resque.git -n resque
git add .; git commit -m "added delayed job recipe"; git push origin master
ey deploy

Manual Installation

Clone/copy this repository into a cookbooks/resque folder (such that you have a cookbooks/resque/recipes/default.rb file). This recipe must be installed as resque and not eycloud-recipe-resque or anything fancy.

Then add the following to cookbooks/main/recipes/default.rb:

require_recipe "resque"

Make sure this and any customizations to the recipe are committed to your own fork of this repository.

Then to upload and apply to EY Cloud for a given environment:

ey recipes upload --apply -e target-environment

Rails 3

To ensure that Resque can discover where Redis installed, create a config/initializers/resque.rb file:

resque_yml = File.expand_path('../../resque.yml', __FILE__)
if File.exist?(resque_yml)
  Resque.redis = YAML.load_file(resque_yml)["redis_uri"]
end

How do I get some Resque workers?

Resque workers can process 1 task/job at a time.

If you have a solo instance, then one Resque worker is provisioned.

If you have one or more app instances, then no Resque workers are provisioned. Instead, add dedicated utility instances to provision Resque workers.

Name your Utility instances resque.

How many Resque workers will I get?

All the available memory of an instance is used for Resque workers, with 300Mb set aside for operating system tasks. Resque workers are allocated 300Mb each.

As above, solo instances of any time have 1 worker.

For utility instances you get approximately the following number of workers:

  • Small -> 4 workers within 1.7 GB
  • Large -> 24 workers within 7.5 GB
  • Extra Large -> 49 workers within 15 GB

  • High Memory Extra Large -> 55 workers within 17.1 GB

  • High-Memory Double Extra Large -> 113 workers within 34.2 GB

  • High-Memory Quadruple Extra Large -> 213 workers within 64.4 GB

  • High-CPU Medium -> 4 workers within 1.7 GB

  • High-CPU Extra Large -> 22 workers within 7 GB

How do I setup Redis?

This recipe assumes that Redis is running within the environment. Fortunately, it is. Redis is automatically running on all Engine Yard Cloud environments.

If you have a dedicate Redis instance called redis then this recipe will discover and use that version of Redis.

How do I use resque-web?

In a Rails 3 app, you can access "resque-web" via your normal Rails app using mounted apps.

As a very simple, moderately secure method, add the following into your routes.rb routes:

require "resque/server"
mount Resque::Server.new, :at => "/resque/34257893542"

For more security, consider the ideas in this gist.

Helpers

  • resque_instances - returns list of instances that have resque running on them
  • resque_instance? - does this instance have resque running on it?