Module: Qtrix

Includes:
Namespacing
Defined in:
lib/qtrix.rb,
lib/qtrix/cli.rb,
lib/qtrix/queue.rb,
lib/qtrix/matrix.rb,
lib/qtrix/version.rb,
lib/qtrix/override.rb,
lib/qtrix/cli/queues.rb,
lib/qtrix/namespacing.rb,
lib/qtrix/matrix/model.rb,
lib/qtrix/cli/overrides.rb,
lib/qtrix/matrix/common.rb,
lib/qtrix/matrix/reader.rb,
lib/qtrix/cli/config_sets.rb,
lib/qtrix/matrix/analyzer.rb,
lib/qtrix/matrix/row_builder.rb,
lib/qtrix/matrix/queue_picker.rb,
lib/qtrix/matrix/queue_prioritizer.rb

Overview

Facade into a dynamically adjusting global worker pool that auto balances workers according to a desired distribution of resources for each queue.

The desired distribution can be modified in real time, and the workers throughout our global pool across all servers should morph to reflect the new desired distribution. Further details on how desired distribution is achieved can be found in the lib/qtrix/matrix.rb comments.

Overrides should be able to be specified, so that we can say out of all of our workers, N should specifically service this list of queues. This is for flood event handling – a queue gets flooded and we need to direct resources to it to help process the jobs faster.

Different configuration sets should be supported, with one being active at a time. A configuration set is a namespaced desired distribution and set of overrides.

This is the primary entry point to the system, a GUI, CLI or script meant to interact with the system should probably work through this module

Defined Under Namespace

Modules: CLI, Matrix, Namespacing Classes: Override, Queue

Constant Summary collapse

VERSION = 
"0.0.1"

Class Method Summary collapse

Methods included from Namespacing

#extract_args, included, #redis, #redis_namespace

Class Method Details

.activate_configuration_set!(namespace) ⇒ Object

Specifies the current configuration set. The namespace must identify a configuration set created with create_configuration_set.



84
85
86
 
# File 'lib/qtrix.rb', line 84

def self.activate_configuration_set!(namespace)
  Namespacing::Manager.instance.change_current_namespace(namespace)
end

.add_override(*args) ⇒ Object

Add a list of queue names to use as an override for a number of worker processes in a configuration set. The number of worker processes will be removed from the desired distribution and start working the list of queues in the override. args should be:

configuration_set: optional, defaults to :current. queues: Array of queue names. processes: Integer specifying the number of workers to override queues for.



126
127
128
129
130
 
# File 'lib/qtrix.rb', line 126

def self.add_override(*args)
  config_set, queues, processes = extract_args(2, *args)
  Qtrix::Override.add(config_set, queues, processes)
  true
end

.clear! ⇒ Object

Clears redis of all information related to the orchestration system



170
171
172
 
# File 'lib/qtrix.rb', line 170

def self.clear!
  Matrix.clear!
end

.configuration_sets ⇒ Object

Returns the list of all configuration sets that have been created



52
53
54
 
# File 'lib/qtrix.rb', line 52

def self.configuration_sets
  Namespacing::Manager.instance.namespaces
end

.connection_config(opts = {}) ⇒ Object

Specifies the redis connection configuration options as per the redis gem.



37
38
39
 
# File 'lib/qtrix.rb', line 37

def self.connection_config(opts={})
  Namespacing::Manager.instance.connection_config(opts)
end

.create_configuration_set(namespace) ⇒ Object

Creates a configuration set for use in the system, which can have its own desired distribution and overrides.



60
61
62
 
# File 'lib/qtrix.rb', line 60

def self.create_configuration_set(namespace)
  Namespacing::Manager.instance.add_namespace(namespace)
end

.current_configuration_set ⇒ Object

Returns the current configuration set in use by the system.



76
77
78
 
# File 'lib/qtrix.rb', line 76

def self.current_configuration_set
  Namespacing::Manager.instance.current_namespace
end

.desired_distribution(config_set = :current) ⇒ Object

Returns a list of objects that define the desired distribution of workers for the current configuration set. Each element will contain the queue name, weight, and resource_percentage (weight / total weight of all queues). By default, this operated on the current configuration set.



95
96
97
 
# File 'lib/qtrix.rb', line 95

def self.desired_distribution(config_set=:current)
  Queue.all_queues(config_set)
end

.map_queue_weights(*args) ⇒ Object

Specifies the queue/weight mapping table for a configuration set. This will be used to generate the queue list for workers and thus the desired distribution of resources to queues. Args can be:

config_set: optional, defaults to current. map: the queue-to-weight mappings as a hash of queue names to

float values.



109
110
111
112
 
# File 'lib/qtrix.rb', line 109

def self.map_queue_weights(*args)
  config_set, map = extract_args(1, *args)
  Qtrix::Queue.map_queue_weights(config_set, map)
end

.operations ⇒ Object

Returns the public operations of the facade. Useful when tinkering in a REPL.



44
45
46
 
# File 'lib/qtrix.rb', line 44

def self.operations
  self.public_methods - Namespacing.methods - Namespacing.instance_methods
end

.overrides(config_set = :current) ⇒ Object

Retrieves all currently defined overrides within a config set. Defaults to use the current config set.



152
153
154
 
# File 'lib/qtrix.rb', line 152

def self.overrides(config_set=:current)
  Qtrix::Override.all(config_set)
end

.queues_for!(hostname, workers) ⇒ Object

Retrieves lists of queues as appropriate to the overall system balance for the number of workers specified for the given hostname.



160
161
162
163
164
165
 
# File 'lib/qtrix.rb', line 160

def self.queues_for!(hostname, workers)
  overrides_queues = Qtrix::Override.overrides_for(hostname, workers)
  delta = workers - overrides_queues.size
  matrix_queues = delta > 0 ? Matrix.queues_for!(hostname, delta) : []
  overrides_queues + matrix_queues.map(&append_orchestrated_flag)
end

.remove_configuration_set!(namespace) ⇒ Object

Removes a configuration set from the system, it will no longer be able to be activated to change the behavior of the system as a whole.



69
70
71
 
# File 'lib/qtrix.rb', line 69

def self.remove_configuration_set!(namespace)
  Namespacing::Manager.instance.remove_namespace!(namespace)
end

.remove_override(*args) ⇒ Object

Removes an override from a current configuration set. That number of worker processes will quit servicing the queues in the override and be brought back into servicing the desired distribution. Args can be:

configuration_set: optional, defaults to :current. queues: Array of queues in the override. processes: Number of processes to remove from overriding.



142
143
144
145
146
 
# File 'lib/qtrix.rb', line 142

def self.remove_override(*args)
  config_set, queues, processes = extract_args(2, *args)
  Qtrix::Override.remove(config_set, queues, processes)
  true
end