resqued - a long-running daemon for resque workers.

Resqued is a multi-process daemon that controls and monitors a pool of resque workers. It works well with slow jobs and continuous delivery.

Installation

Install by adding resqued to your Gemfile

gem 'resqued'

Run resqued with a config file, like this:

resqued config/resqued.rb

Or like this to daemonize it:

resqued -p tmp/pids/resqued-master.pid -D config/resqued.rb

Configuring workers

Let's say you were running workers like this:

rake resque:work QUEUE=high        &
rake resque:work QUEUE=high        &
rake resque:work QUEUE=slow        &
rake resque:work QUEUE=medium      &
rake resque:work QUEUE=medium,low  &

To run the same fleet of workers with resqued, create a config file config/resqued.rb like this:

2.times { worker 'high' }
worker 'slow'
worker 'medium'
worker 'medium', 'low'

Another syntax for workers:

worker_pool 5
queue 'low', :percent => 20
queue 'normal', :percent => 60
queue '*'

This time, you'd end up with something similar to this:

rake resque:work QUEUE=low,normal,* &
rake resque:work QUEUE=normal,*     &
rake resque:work QUEUE=normal,*     &
rake resque:work QUEUE=*            &
rake resque:work QUEUE=*            &

worker and worker_pool accept a hash of options that will be passed to Resqued::Worker. The supported options are:

:interval - The interval to pass to Resque::Worker#run.

Launching in production.

Resqued restarts when its master process receives SIGHUP. It restarts by re-execing the command that you initially ran. There are two main recommendations for running in production.

If you use bundler to install resqued, tell it to generate a binstub for resqued. Invoke this binstub (e.g. bin/resqued) when you start resqued.
Specify a pid file using the -p option. This pidfile will have the PID of the master process. See docs/signals.md for more information about which signals are supported.

If your application is running from a symlinked dir (for example, capistrano's "current" symlink), you'll need to do two more things:

Ensure that your resqued master process is at least 0.7.13 (ps o args= $RESQUED_MASTER_PID should start with "resqued-0.7.13" or higher).
Explicitly set the BUNDLE_GEMFILE environment variable to the symlink dir of your app.
If you're invoking resqued from something that resolves symlinks in pwd, you'll also want to explicitly set the PWD environment variable.

Rolling all of the above advice together, here's a sample that you could use in an upstart script for resqued:

# fragment of /etc/init/resqued.conf

kill signal QUIT

env BUNDLE_GEMFILE=/opt/app/current/Gemfile
env PWD=/opt/app/current
chdir /opt/app/current

exec bin/resqued -c config/resqued.rb -p /opt/app/shared/tmp/pids/resqued.pid

Compatibility with Resque

Resqued does not automatically split comma-separated lists of queues in environment variables like Resque does. To continue using comma-separated lists, split them in your resqued config file:

queue (ENV["QUEUE"] || "*").split(',')

Loading your application

An advantage of using resqued (over rake resque:work) is that you can load your application just once, before forking all the workers.

For a Rails application, you might do this:

before_fork do
  require "./config/environment.rb"
  Rails.application.eager_load!
  # `before_fork` runs in the Listener, and it doesn't actually run any application code.
  ActiveRecord::Base.connection.disconnect!
end

after_fork do |resque_worker|
  # Set up a new connection to the database.
  ActiveRecord::Base.establish_connection
  # `resque_worker.reconnect` already happens
end

Customizing Resque::Worker

You can configure the Resque worker in the after_fork block

after_fork do |resque_worker|
  # Do not fork to `perform` jobs.
  resque_worker.cant_fork = true

  # Wait a loooong time on SIGTERM.
  resque_worker.term_timeout = 1.day

  resque_worker.run_at_exit_hooks = true

  Resque.before_first_fork do
    # ...
  end
end

Full config file example

worker 'high'
worker 'low', :interval => 30

worker_pool 5, :interval => 1
queue 'high', 'almosthigh'
queue 'low', :percent => 20
queue 'normal', :count => 4
queue '*'

before_fork do
  require "./config/environment.rb"
  Rails.application.eager_load!
  ActiveRecord::Base.connection.disconnect!
end

after_fork do |worker|
  ActiveRecord::Base.establish_connection
  worker.term_timeout = 1.minute
end

In this example, a Rails application is being set up with 7 workers:

high
low (interval = 30)
high, almosthigh, low, normal, * (interval = 1)
high, almosthigh, normal, * (interval = 1)
high, almosthigh, normal, * (interval = 1)
high, almosthigh, normal, * (interval = 1)
high, almosthigh, * (interval = 1)

Multiple configurations

If your app has several work machines, each with the same application code but different sets of workers, you might want to have a shared config file for the before_fork and after_fork blocks. You can pass in several config files, and resqued will act as if you concatenated them.

$ resqued config/shared.rb config/pool-a.rb
$ resqued config/shared.rb config/pool-b.rb

Testing

To test your resqued configuration, add a test case like this:

class MyResquedTest < Test::Unit::TestCase
  include Resqued::TestCase
  def test_resqued_config
    assert_resqued 'config/resqued.rb'
  end
end