Karousel

Karousel is an intelligent, client-side job dispenser that accesses a parallelized service without overloading its queue. It dishes out jobs onto a defined number of ‘seats’ on the ‘karousel’, tosses out completed jobs, and refills empty ‘seats’ with new jobs. As a result, the queue on the parallelized service remains lean and clients make responsible use of the service.

Principle of operation

The gem works on a carousel principle. Jobs are loaded onto a carousel-like object. The size of the ‘karousel’ (i.e. its ‘seats’) determines how many objects are active at any given time. For example, if the size of the ‘karousel’ is 20, only 20 objects at a time will access the service.

Karousel loads all jobs, initiates its rotation, and sends each job in succession to a service. For each new job, ‘karousel’ waits for an ‘OK’ from the service, captures the status of the job and optionally stores the response from the service. Starting with the oldest submitted job, ‘karousel’ checks to see if the job is finished. If it is finished, it is processed, removed from the ‘karousel’, and the seat is freed for a new job. After a rotation of the ‘karousel’ empty seats are filled with new jobs and the ‘karousel’ continues to rotate.

Installation

gem install karousel

Usage

Given a Karousel-compatible job class (e.g. KarouselJob) the usage is:

require 'karousel'
karousel = Karousel.new(KarouselJob, 20, 5)
karousel.run

where KarouselJob is the name of your job class, 20 is number of ‘seats’ on your ‘karousel’, and 5 is the time interval in seconds between complete rotations of the ‘karousel’.

You can optionally supply a block to the ‘run’ method to perform your own logging, collect data, or other uses. For example:

count = 0
karousel.run { puts count += 1 }

result = []
karousel.run { result << karousel.cycle_data }

How to write a Job Class

A Job class requires the following signature:

class MyJob < Karousel::ClientJob

  def self.populate(number_of_seats)
    ... #returns an array of the karousel size (number_of_seats would be 20 in our example)
  end

  def send
    ... #sends the job to the service, returns true if it receives an expected response, false otherwise
  end

  def finished?
    ... #returns true if job is done, false otherwise
  end

  def process
    ... #processes the finished job
  end

end

You can also check out github.com/GlobalNamesArchitecture/karousel/blob/master/spec/support/client_job_dummy.rb file or github.com/GlobalNamesArchitecture/karousel_example

Contributing to karousel

Check out the latest master to make sure the feature hasn’t been implemented or the bug hasn’t yet been fixed.
Check out the issue tracker to make sure someone hasn’t already submitted a similar request and/or a solution.
Fork the project.
Start a feature/bugfix branch.
Commit and push until you are happy with your contribution.
Make sure you add tests. This is important so I don’t unintentionally break Karousel in the future.
Please try not to mess with the Rakefile, version, or history. If you want to have your own version, that’s fine, but please isolate to its commit so I can cherry-pick around it.

Copyright

Authors: Dmitry Mozzherin, David Shorthouse