Resque::Forker

Super awesome forking action for Resque workers.

Forking Workers

If you’re like us, you have a sizeable application with many models, libraries and dependencies that are shared between the front-facing UI and the back-end processing. And like us, you’re Resque worker are loading the entire application each time the fire up.

If you’re running 8 workers that can be quite the CPU-churning delay loading them all up. Exactly the problem we’re going to solve by starting the application once and then forking it. Forking all these workers takes milliseconds. Faster restart means faster deploy and less downtime. Yay!

Configuring your Workers to use resque-ctl

Create a Ruby script that loads the applications, handles connections, and decides what kind of workload (how many workers on which queues) to process.

Edit this to your needs and place it in config/resque_workers.rb:

worker_processes 1 # number of worker processes to spawn
worker_queues ["*"] # listen on all worker queues
worker_timeout 30 # timeout a worker
work_interval 3 # intervals to poll

working_directory "/app/path"  # where to run daemonized workers
pid "/app/path/tmp/pids/workers.pid" # master process pid
stderr_path "/app/path/log/workers.stderr.log"
stdout_path "/app/path/log/workers.stdout.log"
preload_app true
daemonize true

GC.respond_to?(:copy_on_write_friendly=) and GC.copy_on_write_friendly = true

setup do|forker|
  defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect!

  if Rails.env.development?
    forker.options.verbose = true
  else
    forker.logger   = Rails.logger
  end

end

# run in master
before_fork do
  defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect!
end

# run in worker
after_fork do
  defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection
end

The configuration should seem very similar to how unicorn sets up it’s configuration.

You can now run workers from the command line:

$ resque-ctl -c config/resque_workers.rb

or with bundler

$ bundle exec resque-ctl -c config/resque_workers.rb

Using a script/workers control file

We’re going to create a Ruby script that loads the applications, handles connections, and decides what kind of workload (how many workers on which queues) to process.

Edit this to your needs and place it in script/workers:

#!/usr/bin/env ruby
require "resque/forker"

# Load the application.
Resque.setup do |forker|
  $:.unshift File.dirname(__FILE__) + "/.." # Makes 1.9.2 happy
  require File.dirname(__FILE__) + "/../config/environment"
  ActiveRecord::Base.connection.disconnect!
  if Rails.env.production?
    forker.logger = Rails.logger
    forker.workload = ["*"] * 4        # 4 workers on all queues
    forker.user "www-data", "www-data" # don't run as root
    forker.options.interval = 1
  else
    forker.options.verbose = true
  end
end
# Stuff to do after forking a worker.
Resque.before_first_fork do
  ActiveRecord::Base.establish_connection
end
Resque.fork!

You can now run workers from the command line:

$ ruby script/workers

In development mode you will get one worker that outputs to the console. In production you get four workers that log messages to the Rails logger and run under the www-data account (never run as root).

Worker processes can’t share connections with each other, so we’re closing the database connection from the master process and then establishing new connection for each individual worker. You’ll have to do the same with other libraries that maintain open connections (MongoMapper, Vanity, etc)

You tell Resque::Forker what workload to process using an array of queue lists. Each array element represents one worker, so 4 elements would start up four workers. The element’s value tell the worker which queues to process. For example, if you want four workers processing the import queue, and two of these workers also processing the export queue:

forker.workload = ["import", "import,export"] * 2

Controlling the Workers

You can use these signals to control individual workers, or send them to the master process, which will propagate them to all workers:

kill -QUIT -- Quit gracefully
kill -TERM -- Terminate immediately
kill -USR1 -- Dump status to syslog
kill -USR2 -- Suspend worker
kill -CONT -- Resume suspended worker

After deploying you want to stop all workers, reload the master process (and the application and its configuration) and have all workers restarted. Simply send it the HUP signal. That easy.

You probably want to suspend/resume (USR2/CONT signals) if you’re doing any maintenance work that may disrupt the workers, like rake db:migrate. Of course you can stop/start the master process, but what would be the fun of that.

Of course, you want the workers to start after reboot and each way to control them. Read on how to use Resque::Forker with Upstart.

Using Upstart

If you’re running a recent release of Ubuntu, you can get Upstart to manage your workers.

Edit this to your needs and place it in /etc/init/workers:

start on runlevel [2345]
stop on runlevel [06]

chdir /var/www/myapp/current
env RAILS_ENV=production
exec script/workers

respawn

After reading this, Upstart to make sure your workers are always up and running. It’s awesome like that.

To start, stop, check status and reload:

$ start workers
$ stop workers
$ status workers
$ reload workers

You need to be root to start/stop the workers. However, if you change ownership of the workers (see fork.user above) you can reload them as that user.

Because of the way Upstart works, there is no need for PID file or running as daemon. Yay for sane process supervisors! When you reload workers, Resque::Forker reloads itself (and the application) while keeping the same PID.

Using Capistrano

Make sure to require “resque/capistrano” at the top of your Capfile and associate worker instances with the roles :worker.

You now have four new task:

workers:pid – Lists PID and additional information for all worker processes.
workers:suspend – Suspends all workers (do not pick any new jobs).
workers:resume – Resumes all workers.
workers:reload – Reloads all workers.

For convenience, the workers:reload task is wired to execute after deploy:restart. In addition, workers:suspend executes before deploy:web:disable, and workers:resume after deploy:web:enable.

Troubleshooting

If you’re using Bundler, you might need to run the script like this:

exec bundle exec script/workers

If you’re using RVM and have a system-wide install, you’ll want to create a wrapper, for example:

$ rvm wrapper 1.9.2 app ruby

Then run the script using the wrapper:

exec app_ruby script/workers

If you’re using RVM and Bundler, don’t forget to uncomment the relevant lines in script/workers.

The point is, when the script starts it will expect both resque and resque-forker must be available for loading (that typically means GEMPATH). Depending on your setup, they may be loaded by Bundler, available in the RVM gemset, installed as system gems, etc.

If you’re hitting a wall, remember that any settings and aliases that you have in .bashrc (RVM, for example, or the path to bundle) are not sourced by Upstart, so commands that “just work” when you run from the console will fail.

What you can do to troubleshoot this situation is run as root in a new shell that doesn’t have your regular account settings:

$ env -i sudo /bin/bash --norc --noprofile

Also, make sure your script is running from the current directory. Say the script path as shown in the Resque console is /var/www/myapp/current/script/worker: reloading this script will load the most recent version of your application.

On the other hand, if the script path looks like /var/www/myapp/releases/20100916, reloading this script will reload the same release over and over.