NAME
bj
SYNOPSIS
bj (migration_code|generate_migration|migrate|setup|plugin|run|submit|list|set|config|pid) [options]+
DESCRIPTION
________________________________
Overview
--------------------------------
Backgroundjob (Bj) is a brain dead simple, zero admin, background priority
queue for Rails. Bj is robust, platform independent (including winblows),
and supports internal or external manangement of the background runner
process.
Jobs can be submitted to the queue directly using the api or from the command
line using the ./script/bj:
api:
Bj.submit 'cat /etc/password'
command line:
bj submit cat /etc/password
Bj's priority queue lives in the database and is therefore durable - your
jobs will live across an app crash or machine reboot. The job management is
comprehensive - capturing stdout, stderr, exit_status, and temporal
statistics about each job:
jobs =
Bj.submit command_or_array_of_commands, :priority => 42
...
jobs.each do |job|
if job.finished?
p job.stdout
p job.stderr
p job.exit_status
p job.started_at
p job.finished_at
end
end
In addition the background runner process logs all commands run and their
exit_status to a log named using the following convention:
rails_root/log/bj.#{ HOSTNAME }.#{ RAILS_ENV }.log
Bj allows you to submit jobs to multiple databases; for instance, if your
application is running in development mode you may do:
Bj.in :production do
Bj.submit 'my_job.exe'
end
Bj manages the ever growing list of jobs ran by automatically archiving them
into another table (by default jobs > 24 hrs old are archived) to prevent the
jobs table from becoming bloated and huge.
All Bj's tables are namespaced and accessible via the Bj module:
Bj.table.job.find(:all) # jobs table
Bj.table.job_archive.find(:all) # archived jobs
Bj.table.config.find(:all) # configuration and runner state
Bj always arranges for submitted jobs to run with a current working
directory of RAILS_ROOT and with the correct RAILS_ENV setting. For
example, if you submit a job in production it will have ENV['RAILS_ENV'] ==
'production' when the job is ran.
When Bj manages the background runner it will never outlive the rails
application - it is started and stopped on demand in parallel with the rails
app parent process. This is also true for ./script/console - Bj will
automatically fire off the background runner to process jobs submitted using
the console iff needed.
Bj starts *one process per server*. For instance, if you are running 3
mongrels via mongrel cluster Bj will automatically fire up *one background
process* per mongrel. This helps you scale background processes in a
relatively POLS fashion, but note that the number of background runners does
not *really* determine throughput - that is determined primarily by the
nature of the jobs themselves and how much work they perform per process,
though having more background processes can also help.
________________________________
Architecture
--------------------------------
If one ignores platform specific details the design of Bj is quite simple: the
main Rails application submits jobs to table, stored in the database. The act
of submitting triggers exactly one of two things to occur:
1) a new long running background runner to be started
2) an existing background runner to be signaled
The background runner refuses to run two copies of itself for a given
hostname/rails_env combination. For example you may only have one background
runner processing jobs on localhost in development mode.
The background runner, under normal circumstances, is managed by Bj itself -
you need do nothing to start, monitor, or stop it - it just works. However,
some people will prefer manage their own background process, see 'External
Runner' section below for more on this.
The runner simply processes each job in a highest priority oldest-in fashion,
capturing stdout, stderr, exit_status, etc. and storing the information back
into the database while logging it's actions. When there are no jobs to run
the runner goes to sleep for 42 seconds; however this sleep is interuptable,
such as when the runner is signaled that a new job has been submitted so,
under normal circumstances there will be zero lag between job submission and
job running for an empty queue.
________________________________
External Runner / Clustering
--------------------------------
For the paranoid control freaks out there (myself included) it is quite
possible to manage and monitor the runner process manually. This can be
desirable in production setups where monitoring software may kill leaking
rails apps periodically or in any case where you want to montior and control
the background process directly.
Recalling that Bj will only allow one copy of itself to process jobs per
hostname/rails_env pair we can simply do something like this in cron
cmd = bj run --forever \
--rails_env=development \
--rails_root=/Users/ahoward/rails_root
*/15 * * * * $cmd
this will simply attempt the start the background runner every 15 minutes if,
and only if, it's not *already* running.
In addtion to this you'll want to tell Bj not to manage the runner itself
using
Bj.config["production.no_tickle"] = true
For instance in ./config/initializer/bj.rb or ./config/environment.rb.
Note that, for clusting setups, it's as simple as adding a crontab and
config entry like this for each host you app is running on. Because Bj
throttles background runners per hostname this will allow one runner per
hostname - making it quite simple to cluster three nodes behind a besieged
rails application.
________________________________
Designing Jobs
--------------------------------
Bj runs it's jobs as command line applications. It ensures that all jobs run
in RAILS_ROOT so it's quite natural to apply a pattern such as
mkdir ./jobs
edit ./jobs/background_job_to_run
...
Bj.submit "./jobs/background_job_to_run"
If you need to run you jobs under an entire rails environment you'll need to
do this:
Bj.submit "./script/runner ./jobs/background_job_to_run"
Obviously "./script/runner" loads the rails environment for you. It's worth
noting that this happens for each job and that this is by design: the reason
is that most rails applications leak memory like a sieve so, if one were to
spawn a long running process that used the application code base you'd have a
lovely doubling of memory usage on you app servers. Although loading the
rails environment for each background job requires a little time, a little
cpu, and a lot less memory. A future version of Bj will provide a way to load
the rails environment once and to process background jobs in this environment,
but anyone wanting to use this in production will be required to duct tape
their entire chest and have a team of oxen rip off the tape without screaming
to prove steelyness of spirit and profound understanding of the other side.
Don't forget that you can submit jobs with command line arguments:
Bj.submit "./jobs/a.rb 1 foobar --force"
and that you can do powerful things by passing stdin to a job that powers
through a list of work. For instance, assume a "./jobs/bulkmail" job
resembling
STDIN.each do |line|
address = line.strip
mail_message_to address
end
then you could
stdin = [
"[email protected]",
"[email protected]",
"[email protected]",
]
Bj.submit "./script/runner ./jobs/bulkmail", :stdin => stdin
and all those emails would be sent in the background.
Bj's power is putting jobs in the background in a simple and robust fashion.
It's your task to build intelligent jobs that leverage batch processing, and
other, possibilities. The upshot of building tasks this way is that they are
quite easy to test before submitting them from inside your application.
________________________________
Install
--------------------------------
Bj can be installed two ways: as a plugin or via rubygems
plugin:
1) ./script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
2) ./script/bj setup
gem:
1) $sudo gem install bj
2) add "require 'bj'" to config/environment.rb
3) bj setup
________________________________
Api
--------------------------------
submit jobs for background processing. 'jobs' can be a string or array of
strings. options are applied to each job in the 'jobs', and the list of
submitted jobs is always returned. options (string or symbol) can be
:rails_env => production|development|key_in_database_yml
when given this keyword causes bj to submit jobs to the
specified database. default is RAILS_ENV.
:priority => any number, including negative ones. default is zero.
:tag => a tag added to the job. simply makes searching easier.
:env => a hash specifying any additional environment vars the background
process should have.
:stdin => any stdin the background process should have. must respond_to
to_s
eg:
jobs = Bj.submit 'echo foobar', :tag => 'simple job'
jobs = Bj.submit '/bin/cat', :stdin => 'in the hat', :priority => 42
jobs = Bj.submit './script/runner ./scripts/a.rb', :rails_env => 'production'
jobs = Bj.submit './script/runner /dev/stdin',
:stdin => 'p RAILS_ENV',
:tag => 'dynamic ruby code'
jobs Bj.submit array_of_commands, :priority => 451
when jobs are run, they are run in RAILS_ROOT. various attributes are
available *only* once the job has finished. you can check whether or not a
job is finished by using the #finished method, which simple does a reload and
checks to see if the exit_status is non-nil.
eg:
jobs = Bj.submit list_of_jobs, :tag => 'important'
...
jobs.each do |job|
if job.finished?
p job.exit_status
p job.stdout
p job.stderr
end
end
See lib/bj/api.rb for more details.
________________________________
Sponsors
--------------------------------
http://quintess.com/
http://www.engineyard.com/
http://igicom.com/
http://eparklabs.com/
http://your_company.com/ <<-- (targeted marketing aimed at *you*)
________________________________
Version
--------------------------------
1.0.1
PARAMETERS
--rails_root=rails_root, -R (0 ~> rails_root=)
the rails_root will be guessed unless you set this
--rails_env=rails_env, -E (0 ~> rails_env=development)
set the rails_env
--log=log, -l (0 ~> log=STDERR)
set the logfile
--help, -h
AUTHOR
ara.t.howard@gmail.com
URIS
http://codeforpeople.com/lib/ruby/
http://rubyforge.org/projects/codeforpeople/
http://codeforpeople.rubyforge.org/svn/rails/plugins/