Pipeline

Description

Pipeline is a Rails plugin/gem to run asynchronous processes in a configurable pipeline.

Features

  • Execution of sequential user-defined stages in an asynchronous pipeline

  • Persistence of pipeline instances and stages

  • Error recovery strategies:

    • Irrecoverable errors fail the entire pipeline

    • Recoverable errors are automatically retried (using dj’s exponential retry strategy)

    • Recoverable errors that require user input pause the pipeline for further retry

  • Cancelling of a paused pipeline

Known Issues:

  • Rails’ tests/specs will only work if delayed_job is installed as a gem, rather than a plugin.

Installation

Add the following lines to your config/environment.rb file:

config.gem "pipeline"

Run the following:

rake gems:install
rake gems:unpack # Optional, if you want to vendor the gem
script/generate pipeline # To generate the migration scripts that will store pipelines
rake db:migrate

You will also need to run your Delayed Job workers that will process the pipeline jobs in the background:

rake jobs:work

Dependencies

Usage

Check examples for more examples (including error-recovery and cancelling)

class Step1 < Pipeline::Stage::Base
  def perform
    puts("Started step 1")
    sleep 2
    puts("Finished step 1")
  end
end

class Step2 < Pipeline::Stage::Base
  def perform
    puts("Started step 2")
    sleep 3
    puts("Finished step 2")
  end
end

class TwoStepPipeline < Pipeline::Base
  define_stages Step1 >> Step2
end

Pipeline.start(TwoStepPipeline.new)

Copyright © 2009 Danilo Sato. See LICENSE for details.