Ruby Reduce

Ruby Reduce is a small library designed to run reduce commands on a Rails 3.x log without needing to setup a hadoop cluster to do it. So it is basically designed to be a small scale reduce function. The original idea was to make the gem plugable so that you could plugin different inputers to chunk non Rails 3 log files. Also there was supposed to be a similar plugin design for the output but that has not happened yet. So currently it only accepts rails 3 files for input and writes the results to MongoDB.

Installation

gem install ruby_reduce

Usage

The usage is also very straight forward and demostraited by this code snippet.

require 'date' #only required because I parse a date
require 'ruby_reduce' # 1) Require ruby reduce

module RubyReduce
  #return host and port and mongo_db (or nothing for default host and port)
  def self.mongo_connection
  end

  #actual mongo db to connect to
  def self.mongo_db
    return 'test_db'
  end
end

#actually running the reduce function
r =  RubyReduce.reduce({
    :input => '../production.log', #the location of the log file to reduce
    # The map function can be anything that response to call and accepts 2 paramaters.  
    # The first paramater is the log file name and the second are all the log statements 
    # for a single rails request.  The only thing required here is that you **emit** a 
    # key value pair to reduce later.  simple call the `emit(key, value)` function to accomplish this.
    :map => Proc.new do |key, value|
      processed_by = /Processing by (\w*#\w*)/.match(value)

        unless processed_by.nil?
          emit processed_by[1].gsub!("#", "_"), value
        end
      end,
    # The reduce function can be anything that responds to call and takes one argument.  
    # This argument will be the value emited in the map function (so reduce is called once 
    # for every key value pair emited.  Also in the reduce function you need to **emit** 
    # the result of the reduce.  Simply call `emit(result)` to tell RubyReduce what the result is.
    :reduce => Proc.new do |log_statement|
        date = /Started [GET|POST|DELETE|PUT|HEAD].* at (.*)/.match(log_statement) 
        unless date.nil? 
          processing_time = /Completed \d* .* in (\d*)/.match(log_statement)[1]
          emit({'date' => Time.parse(date[1]), 'processed_time' => processing_time})
        end
      end,
    #MongoDB collection name to write results to
    :output => 'graph_data'
  })

Once all of the key value pairs have been reduced the results will be collected by key and written into mongodb with one document for each key emited in the map function. The id (_id) of this document is the emited key

Limitations

Right now this is at best alpha software. There are no tests and while I use it for my on projects it is not really been on a wide range of problems. Your feedback is welcome.

Questions

Contact me at joshsmoore@gmail.com with question, comments, or just that you are using the library and want me to continue work on it.