Module: MapRedus

Includes:
RedisSupport
Defined in:
lib/mapredus.rb,
lib/mapredus/keys.rb,
lib/mapredus/mapper.rb,
lib/mapredus/master.rb,
lib/mapredus/process.rb,
lib/mapredus/reducer.rb,
lib/mapredus/support.rb,
lib/mapredus/inputter.rb,
lib/mapredus/finalizer.rb,
lib/mapredus/outputter.rb,
lib/mapredus/filesystem.rb,
lib/mapredus/default_classes.rb

Defined Under Namespace

Modules: Support Classes: Adder, Counter, FileSystem, Finalizer, Helper, Identity, InputStream, InvalidInputStream, InvalidMapper, InvalidProcess, InvalidReducer, JsonOutputter, Mapper, Master, Outputter, Process, ProcessSpecificationError, QueueProcess, RecoverableFail, RedisHasher, Reducer, ToRedisHash, WordCounter, WordStream

Constant Summary collapse

RedisKey = 
MapRedus::Keys
ProcessInfo = 
RedisKey
DEFAULT_RESULT_KEY =

The default location for the result

"mapredus:process:PID:result"
NOT_STARTED =

Use these constants to keep track of the progress of a process

Example

state => map_in_progress
         reduce_in_progress
         finalize_in_progress
         complete
         failed
         not_started

contained in the ProcessInfo hash (redis_key :state, “mapredus:process:PID:master:state”)

"not_started"
INPUT_MAP_IN_PROGRESS = 
"mappers"
REDUCE_IN_PROGRESS = 
"reducers"
FINALIZER_IN_PROGRESS = 
"finalizer"
COMPLETE = 
"complete"
FAILED = 
"failed"
STATE_MACHINE = 
{ nil => NOT_STARTED,
NOT_STARTED => INPUT_MAP_IN_PROGRESS,
INPUT_MAP_IN_PROGRESS => REDUCE_IN_PROGRESS,
REDUCE_IN_PROGRESS => FINALIZER_IN_PROGRESS,
FINALIZER_IN_PROGRESS => COMPLETE}

Instance Method Summary collapse

Instance Method Details

#register_mapper(klass) ⇒ Object 



52
 
# File 'lib/mapredus.rb', line 52

def register_mapper(klass); end

#register_reducer(klass) ⇒ Object

TODO: When you send work to a worker using a mapper you define, the worker won’t have that class name defined, unless it was started up with the class loaded



51
 
# File 'lib/mapredus.rb', line 51

def register_reducer(klass); end