Fire & Forget

Fire & Forget is a simple, framework agnostic, background task launcher for Ruby web apps.

What it does:

Designed for infrequent calls to very long running operations
Provides a mechanism for calling any external script from within a HTTP request
Gives simple inter-process communication for status updates
Launches long running tasks as completely independent processes so that you can restart/redeploy your application without interrupting any running tasks
Provides a ‘kill’ mechanism so you can cancel your tasks

What it doesn’t:

Have any kind of queue mechanism. If you fire the same task twice then that task will be running twice
Have any persistant state
Work in non-UNIX environments

Quick Start

gem install fire_and_forget

FAF works by running a simple UNIX socket server so before we want to use it we have to start this server.

$ fire_forget
FAF process 16235 listening on /tmp/fire_and_forget.sock ...

To change the path of the socket file use the ‘-s’ parameter:

$ fire_forget -s /path/to/socket
FAF process 16240 listening on /path/to/socket ...

If you do this you will need to set the web-app to use the right values:

FireAndForget.socket = "/path/to/socket"

Now inside our Ruby app:

require 'rubygems'
require 'fire_and_forget'

# First set register our task by giving it a name and a path to a script file
FireAndForget.add_task(:long_running_task, "/path/to/script")
# => #<FireAndForget::TaskDescription @name=:long_running_task ...>

# when we want to call the script we simple call #fire passing the name of the
# task and the options we want to pass
FireAndForget.fire(:long_running_task, {:param3 => "value3"})
# => "OK"

# Or, use the name as a method:
FireAndForget.long_running_task({:param3 => "value3"})
# => "OK"

This will result in the following command being exec’d in an independent process:

/path/to/script --param3="value3"

It up to the script to parse and deal with the command line options.

The options for the #add_task call are as follows:

task_name: A symbol giving the (unique) name for this task
path_to_binary: The path on disk of the executable to run when this task is launched
niceness: an integer in the range 0-20 giving the 'niceness' of the task process.
default_params: Command line parameters to pass to each invocation of this task
env: settings to add to this task's environment

The higher the nice value, the lower the priority of the task. Setting a high ‘nice’ allows you to launch intensive background tasks without affecting the performance of your front line HTTP services.

For example:

FireAndForget.add_task(:long_running_task, "/path/to/script", 12,
  {:param1 => "value1", :param2 => "value2", :param3 => "value3"},
  {"ENVIRONMENT_SETTING" => "Something"}
)

this will add some default parameters to all invocations of the task, so now the call

FireAndForget.fire(:long_running_task, {:param3 => "newvalue3"})

will issue the following command in the background

/path/to/script --param1="value1" --param2="value2" --param3="newvalue3"

as well as setting it’s nice value to 12 and updating ENV with the extra parameter “ENVIRONMENT_SETTING”.

Parameter values are encoded in a format compatible with Thor:

Arrays are encoded as –array=1 2 3 Hashes as –hash=key1:value1 key2:value2

Again, it is up to the script to decode these values.

Interprocess communication is relatively easy. Take the following as the source of the script “/path/to/script”

#!/usr/bin/env ruby

require 'rubygems'
require 'fire_and_forget'

# this will mix in the F&F status methods
# the F&F configuration (socket and task-name) is calculated automatically from ENV parameters
include FireAndForget::Daemon

30.times do |i|
  # update our status. What you put here is up to you, but it should be a String
  set_task_status("#{i+1} of 30")
  sleep(1)
end

Now in the client all we have to do to get the status for our task is:

FireAndForget.get_status(:long_running_task)
# => "18 of 30"

If we decided we’ve had enough then we can kill it:

# Send "SIG_TERM" to our task
FireAndForget.term(:long_running_task)

# Or send any signal (see the Process.kill documentation)
FireAndForget.kill(:long_running_task, "HUP")

Supporting multiple users

Tasks will attempt to run as the same user as the originating process. To enable this, and hence enable a single F&F server to launch tasks for any number of separate web-apps, simply run the F&F server as root. See the following section on Security for the implications of this and how to stop unlimited access to the service.

Security

F&F is intended to be run in a relatively trusted environment. You the UNIX socket file has its permissions set so that only the process owner and a defined group can write to the file. To change the UNIX group that has access pass the group name as a parameter to the server:

$ fire_forget -s /path/to/socket -g webapp_group

Only members of the ‘webapp_group’ will be able to launch processes through the server. Additionally, only scripts belonging to the same user as the F&F task description are executed.

In the meantime, as I say, this is defined to be run in a trusted environment, for instance on a single non-shared server (potentially running many sites).

TODO

Improve security restrictions (perhaps by limiting the tasks that each user can launch)
Improve the tests so they are closer to testing the full stack
Add in scheduled/delayed execution of tasks

Contributing to fire_and_forget