# Requirements
RabbitMQ, MYSQL with one user and one database
gem install amqp gem install json gem install data_mapper gem install dm-myql-adapter gem install dm-migrations gem install zip gem install mb-aws-s3
For debug, one can use dm-sqlite-adapter. SQLite does not fit for production and concurrency issues may occur. To avoid this, only 1 workflow handler and 1 job handler should be used. MySQL handles correctly the concurrency with lock mechanisms.
# Installation
Export or clone repository [email protected]:osallou/aws-s3.git Copy lib/aws directory to manband lib directory (lib/manband/aws) S3 host information can be set via jobhandler command-line
# Configuration
Some configuration can be set via a config file with the –conf option. Program will also search for a file “.manband” in user home directory. If none is used, program will use defaults. An example conf file is available in test directory (conf.yaml).
For storage in S3, user credentials must be stored in database. If user has access to the database (non shared install), the userband script will help setting up the user (see User Management). In a shared install, one MUST use the web interface to setup credentials and launch workflows as there is no control on user id.
# Security
Each workflow is ran in a specific set of queues in a virtual host in RabbitMQ. This access is controlled via AMQP_URL credentials, restricting access only to a user or a set of users. Scripts using database access (manband, userband) must not be used in a shared install. One should use remoteband to launch workflows (warning: workflow file must be accessible by nodes). Access to user credentials (s3,..) should be allowed only via the web interface or a custom interface.
# Execution
Before executing the programs, one must set in environment:
export AMQP_URL=“amqp://johndoe:abc123@localhost/my_vhost” export MYSQL_URL=“mysql://USER:PASSWORD@localhost/MYDB”
All programs, including the web interface need access to the database and RabbitMQ. In a cloud environment, where each user or group of user have to launch his own instances, two solutions: 1) A master instance holds the web interface and the database. The master instance could then launch new instances (workflowhandler or jobhandler) with MYSQL_URL refering to master instance 2) For each user, provide with the AMQP queue a dedicated MySQL database. Then user can refer to this database for his handlers.
In all cases, cloud instances would need to get at boot time those 2 variables before executing the handlers or the web interface.
## Workflow Handler
It is possible to run multiple workflow handler to handler the message load
ruby -rubygems workflowhandler.rb
Use -d for message tracking.
## Job Handler
It is possible to run multiple job handlers to handler job commands
ruby -rubygems jobhandler.rb -i myhandlername
## Main program
For usage:
ruby -rubygems manband -h
To start a new workflow:
ruby manband.rb -w test/samplew.yaml
If user do not have MYSQL credentials, user can use:
ruby remoteband.rb -w test/samplew.yaml
remoteband initiate a “new” message to start a workflow without MYSQL credentials need.
## Web interface
For production mode, on port 4567:
rackup -E production -p 4567
## User management
By default, admin account has password “admin”. This should be modified after installation.
userband.rb can help to add or update users without the web interface
## S3 storage
By default, bucket used is “manband”. It can be specified in command-line.
## Gem creation
Execute:
rake package
Gem is created in pkg directory Webband may use this gem, or use current files with same hierarchy
## History
0.8: Add Interactive feature 0.7:
Add IF management
Fix some dispatching issues
0.6.1: Increase file path max size 0.6.0: Remove mb-minion dependency. New bunny gem is incompatible now, code has been rewriten to remove the dependency and use only amqp.