blue_colr, database-based process launcher

Overview

blue_colr allows you to easily launch processes using database as a queue. It consists of bluecolrd, a deamon that executes whatever finds in a queue, and a DSL for enqueuing processes that enables you to easily describe the order and dependencies of processes.

Installation

gem install blue_colr

You may want to install log4r gem as well, as it provides more powerful logging features than builtin Ruby’s Logger.

Example

require 'blue_colr'

BlueColr.launch do
  run 'echo These processes'
  run 'echo will be ran sequentially.'
  parallel do
    run 'echo And these'
    sequential do
      run 'echo (but not'
      run 'echo these two)'
    end
    run 'echo in parallel.'
  end
  run 'echo These will execute'
  run 'echo after all above are finished.'
end

Previous code will queue processes within the database, keeping them in dependency order. Those within sequential block (and in root block, by default) will run each after the one before finishes. Those within parallel block will run in parallel. The commands after parallel block will be executed after all the commands in parallel block are sucessfully finished.

Note: the code above will not start the processes by itself, but enqueue them to the database, by default. A separate process called bluecolrd is used for that.

The following chart, generated by the same code above, is its execution sequence:

Requirements and Configuration

In order to access the database, blue_colr requires sequel ORM library, if you don’t have it, its gem will be installed along with blue_colr.

Blue_colr uses a relational database to simulate a process queue so you will have to provide one. It relies on two tables, named process_items and process_item_dependencies to work. db/ directory contains Sequel migrations for creating these two:

sequel -m db/ sqlite://examples/test.db

Basic configuration is passed to blue_colr either by setting options from your code, or (if not set), blue_colr will parse your command line arguments and get the path to yaml configuration file, using -c option.

bluecolrd

Blue_colr daemon is constantly running, checking the database for newly enqueued processes, and executing them in a subshell, observing the order.

bcrun

This script is used to launch arbitrary command through blue_colr. You might want to do that if you want to keep track of the stuff you launch (as everything goes through a database table).

bcrun -c path_to_config.yaml -x "command to execute"

Environments

An environment is something like category which you assign to a set of processes when enqueuing them. Then you can have multiple daemons running, each one of them targeting specific environment. That allows easy distribution of your tasks across multiple machines, while keeping them synchronized, like the following scenario:

• Start tasks a and b on machine X and c on machine Y

• When all above are sucessfully done, start task d on machine Z

ToDo

• More tests

• Better docs

• Examples