Module: CloudCrowd

Defined in:
lib/cloud-crowd.rb,
lib/cloud_crowd/node.rb,
lib/cloud_crowd/action.rb,
lib/cloud_crowd/models.rb,
lib/cloud_crowd/server.rb,
lib/cloud_crowd/worker.rb,
lib/cloud_crowd/helpers.rb,
lib/cloud_crowd/inflector.rb,
lib/cloud_crowd/exceptions.rb,
lib/cloud_crowd/models/job.rb,
lib/cloud_crowd/asset_store.rb,
lib/cloud_crowd/command_line.rb,
lib/cloud_crowd/models/work_unit.rb,
lib/cloud_crowd/helpers/resources.rb,
lib/cloud_crowd/models/node_record.rb,
lib/cloud_crowd/asset_store/s3_store.rb,
lib/cloud_crowd/helpers/authorization.rb,
lib/cloud_crowd/asset_store/filesystem_store.rb

Defined Under Namespace

Modules: Helpers, Inflector, ModelStatus Classes: Action, AssetStore, CommandLine, Error, Job, Node, NodeRecord, Server, WorkUnit, Worker

Constant Summary collapse

VERSION =

Keep this version in sync with the gemspec.

'0.2.1'
SCHEMA_VERSION =

Increment the schema version when there’s a backwards incompatible change.

3
ROOT =

Root directory of the CloudCrowd gem.

File.expand_path(File.dirname(__FILE__) + '/..')
PROCESSING =

A Job is processing if its WorkUnits are in the queue to be handled by nodes.

1
SUCCEEDED =

A Job has succeeded if all of its WorkUnits have finished successfully.

2
FAILED =

A Job has failed if even a single one of its WorkUnits has failed (they may be attempted multiple times on failure, however).

3
SPLITTING =

A Job is splitting if it’s in the process of dividing its inputs up into multiple WorkUnits.

4
MERGING =

A Job is merging if it’s busy collecting all of its successful WorkUnits back together into the final result.

5
COMPLETE =

A Job is considered to be complete if it succeeded or if it failed.

[SUCCEEDED, FAILED]
INCOMPLETE =

A Job is considered incomplete if it’s being processed, split up or merged.

[PROCESSING, SPLITTING, MERGING]
DISPLAY_STATUS_MAP =

Mapping of statuses to their display strings.

['unknown', 'processing', 'succeeded', 'failed', 'splitting', 'merging']

Class Attribute Summary collapse

Class Method Summary collapse

Class Attribute Details

.config ⇒ Object (readonly)

Returns the value of attribute config.



82
83
84
 
# File 'lib/cloud-crowd.rb', line 82

def config
  @config
end

Class Method Details

.actions ⇒ Object

CloudCrowd::Actions are requested dynamically by name. Access them through this actions property, which behaves like a hash. At load time, we load all installed Actions and CloudCrowd’s default Actions into it. If you wish to have certain nodes be specialized to only handle certain Actions, then install only those into the actions directory.



134
135
136
137
138
139
140
141
142
143
144
145
146
 
# File 'lib/cloud-crowd.rb', line 134

def actions
  return @actions if @actions
  @actions = {}
  default_actions   = Dir["#{ROOT}/actions/*.rb"]
  installed_actions = Dir["#{@config_path}/actions/*.rb"]
  custom_actions    = Dir["#{CloudCrowd.config[:actions_path]}/*.rb"]
  (default_actions + installed_actions + custom_actions).each do |path|
    name = File.basename(path, File.extname(path))
    require path
    @actions[name] = Module.const_get(Inflector.camelize(name))
  end
  @actions
end

.central_server ⇒ Object

Get a reference to the central server, including authentication if configured.



106
107
108
 
# File 'lib/cloud-crowd.rb', line 106

def central_server
  @central_server ||= RestClient::Resource.new(CloudCrowd.config[:central_server], CloudCrowd.client_options)
end

.client_options ⇒ Object

The standard RestClient options for the central server talking to nodes, as well as the other way around. There’s a timeout of 5 seconds to open a connection, and a timeout of 30 to finish reading it.



113
114
115
116
117
118
119
120
121
 
# File 'lib/cloud-crowd.rb', line 113

def client_options
  return @client_options if @client_options
  @client_options = {:timeout => 30, :open_timeout => 5}
  if CloudCrowd.config[:http_authentication]
    @client_options[:user]      = CloudCrowd.config[:login]
    @client_options[:password]  = CloudCrowd.config[:password]
  end
  @client_options
end

.configure(config_path) ⇒ Object

Configure CloudCrowd by passing in the path to config.yml.



85
86
87
88
 
# File 'lib/cloud-crowd.rb', line 85

def configure(config_path)
  @config_path = File.expand_path(File.dirname(config_path))
  @config = YAML.load_file(config_path)
end

.configure_database(config_path, validate_schema = true) ⇒ Object

Configure the CloudCrowd central database (and connect to it), by passing in a path to database.yml. The file should use the standard ActiveRecord connection format.



93
94
95
96
97
98
99
100
101
102
 
# File 'lib/cloud-crowd.rb', line 93

def configure_database(config_path, validate_schema=true)
  configuration = YAML.load_file(config_path)
  ActiveRecord::Base.establish_connection(configuration)
  if validate_schema
    version = ActiveRecord::Base.connection.select_values('select max(version) from schema_migrations').first.to_i
    return true if version == SCHEMA_VERSION
    puts "Your database schema is out of date. Please use `crowd load_schema` to update it. This will wipe all the tables, so make sure that your jobs have a chance to finish first.\nexiting..."
    exit
  end
end

.display_status(status) ⇒ Object

Return the displayable status name of an internal CloudCrowd status number. (See the above constants).



125
126
127
 
# File 'lib/cloud-crowd.rb', line 125

def display_status(status)
  DISPLAY_STATUS_MAP[status] || 'unknown'
end