Qujo

The basic idea behind Qujo came from a simple idea:

Store information about jobs in my application domain database, rather than in Redis

Qujo wraps a Resque worker with a model class and some basic intelligence around executing jobs and storing their output. It also provides a logger for Jobs, retry functionality, Rails integration, Bootstrap integration, polling javascript for displaying the running jobs and workers in a page within your app.

Databases

Currently, Qujo only works for Mongoid. It doesn't lend well to ActiveRecord-based models, as it talks to the database a lot (probably more than it should).

Queue support

I've started some work seeing if I can get Qujo to work with Sidekiq, but I'm not sure it's worth the effort. Sidekiq stands much better on it's own, having solved some of the problems that I experienced with Resque. Though you still have to look somewhere else for errors and stack traces.

Setup

Add Qujo to your Gemfile

gem 'qujo'

Create an initializer for Rails

# config/initializers/qujo.rb
require 'qujo/queue/resque'     # we're using Resque
require 'qujo/database/mongoid' # we're using Mongoid

Qujo.configure do |config|
  # use Yell for our logger
  config.logger = Yell.new do |l|
    l.level = [:info, :warn, :error, :fatal]
    l.adapter :file, File.join(Rails.root, "log", "qujo.log")
  end
end

Create a Job model

The following Job model is the parent of all Jobs that do the actual work.

# app/models/job.rb
class Job
  include Qujo::Database::Mongoid

  include Qujo::Queue::Resque

  include Qujo::Concerns::Common
  include Qujo::Concerns::Logging
  include Qujo::Concerns::Status

  # any custom functionality shared for all jobs

end

Mount the Qujo engine

# config/routes.rb
MyApp::Application.routes.draw do
  # ... normal routes

  mount Qujo::Engine => "/"
end

After this is configured, you can view the jobs at localhost:3000/jobs.

Bootstrap

If you're using bootstrap, there is a Qujo template to integrate some status information into your bootstrap navigation.

<header class="navbar navbar-fixed-top">
  <nav class="navbar-inner">
    <div class="container">
    <!-- ... -->
    <ul class="nav pull-right">
      <!-- ... -->
      <li>
        <%= render 'qujo/common/bootstrap', jobs: {header: ["Jobs::Class::One", "Jobs::Class::Two"], background: ["Jobs::Class::Three"]} %>
      </li>
    </ul>
    <!-- ... -->

The data you pass to jobs should be a hash of keys and arrays. The keys are the header names, the arrays are a list of strings of Job class names.

When this data is configured, you can kick off jobs from the UI, just by clicking on their link in the dropdown.

Color codes:

black

making request to server (normally it will blink black every 2 seconds)

gray

normal - things are good

errors - job with error, resque has failed jobs, or can't talk to resque (workers aren't running)

orange

after 10 minutes, the page will stop polling, just to lower the traffic to the server and not fill logs with polling

Create your first Job

This job inherits from the previous Job parent class and defines a work method.

# app/models/jobs/hello/world.rb
class Jobs::Hello::World < Job
  # the work method is where the work goes
  def work
    info "Hello world!"
  end
end

Integrate with your other models

Imagine that you have a model in your app that tracks purchases. You want to let the customer know that the purchase has succeeded.

Create a Job class named Jobs::Purchase::Notify

# app/models/jobs/purchase/notify.rb
class Jobs::Purchase::Notify < Job
  def work
    # notify the customer
    purchase = model # model is a helper to grab the associated model
    purchase.do_notify
  end
end

Now if you want to queue that job to run, all you need to do (from the controller, or another job)

purchase.enqueue(:notify, optionshash)

Qujo will take care of the rest.

How Qujo does this, it figures out the job class like so: Jobs::<Model>::<Action>

In the above example, notify is the action, so it translates this to Jobs::Purchase::Notify, creates an instance, stores it in the database and tells Resque to enqueue it.

Passing options to jobs

purchase.enqueue(:notify, optionshash)

In the above, optionshash is just a hash of any data that you want to pass into the Job. It is accessible from within the job with the data helper.

Keep in mind, because of the way that Resque stores (marshalls and unmarshalls) data in Redis, all keys will be converted to strings.

# either
purchase.enqueue(:notify, :blarg => true)

# or this
purchase.enqueue(:notify, blarg: true)

# or this
purchase.enqueue(:notify, {"blarg" => true})
# or any other hash

# will result in:
data = {
  "blarg" => true
}

The views

By default, Qujo provides some views and routes to handle querying the job models.

Qujo sugar

Wait

A method to have a job wait while the return value is true. I've considered inversing this, as it probably makes more sense to wait while false.

# app/models/jobs/purchase/notify.rb
class Jobs::Purchase::Notify < Job
  def work
    # notify the customer
    purchase = model # model is a helper to grab the associated model

    # this could potentially be calling another job that sets
    # email#sent to true once it has completed successfully
    email = purchase.do_notify

    # email#sent returns true once the email has been delivered
    wait do
      ! email.sent
    end
  end
end

The wait method supports a few options

interval: integer - the length of time to sleep in seconds for each loop (default: 3)
maximum: integer - the maximum time in seconds to wait (default: 600)

Keep in mind, that right now jobs are not re-entrant, so when a job is waiting, the worker is doing nothing. If you have a lot of jobs that wait, you could end up having all of your workers waiting for something to complete.

I have a plan to change this functionality to work around the re-entrancy problem.

Parent jobs

Qujo can also store parent jobs, when jobs need to run a set of things in order. The API for this is still being worked out.

Workflow

Some basic workflow handling. A job transitions through the following states:

new -> working -> complete
               -> error -> retry => working ...

The name

Pronunciation kyoo'-joe

Naming things is hard. Coming up with something clever, and finding it unused in the community is even harder.

Qujo's name comes from a combination of the words queue and job.

queuejob
quejob
qujob
qujo
profit!

TODO

document: qujo/resque wrapper page
document: better documentation around wait functionality.
build out parent -> child job relationships
rails generator
Smarter workflows, use workflow gem
Job re-entrancy. When a job waits, just schedule it 5 seconds from now and pick up where it left off.
Sidekiq support?
ActiveRecord support?

Wanna help?

Fork me
Clone me
Branch me
Change me
Push me
"Submit a Pull Request" me :P

License

MIT-LICENSE