SBM - Simple Batch Manager
Manages running / coordinating batch processes running across multiple hosts.
Uses redis as a simple coordinator to ensure split work runs across hosts evenly.
Note: SBM is still a hack. It's untested. Don't use this for production stuff. Please!
Installation
Add this line to your application's Gemfile:
gem 'sbm'
And then execute:
$ bundle
Or install it yourself as:
$ gem install sbm
Usage
SBM is composed of a set of simple scripts useful for running in shells. The only required
variable is SBM_WORKER
as an environment variable - the name of the node the current job is
running on.`
#!/usr/bin/env bash -e
export SBM_WORKER="$(hostname)-$$"
# Used if you have a bunch of different batches with the same name:
# export SBM_COORDINATOR='your-groups'
sbm start-work my-test-batch
rake do:your:work
sbm complete-work my-test-batch && sbm wait-for my-test-batch 20 # There are 20 nodes running this process
sbm status
Wait for simply checks the number of items in the completed set have the correct length.
Please note that by default it uses redis for this, so to change your default redis use REDIS_URI
.
Commands
sbm status
- Show status of all batches.sbm start-batch batch-name
- Current worker starts the specified batch.sbm complete-batch batch-name
- Current worker starts the specified batch.sbm wait-for batch-name count
- Wait untilcount
workers have completedbatch-name
.sbm clear-batch batch-name
- Clear information for the given batch.sbm clear-batches
- Clear all batch info.sbm clear-workers
- Clear all worker info.
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request