Class: RequestLogAnalyzer::Controller

Inherits:
Object
  • Object
show all
Defined in:
lib/request_log_analyzer/controller.rb

Overview

The RequestLogAnalyzer::Controller class creates a LogParser instance for the requested file format and connect it with sources and aggregators.

Sources are streams or files from which the requests will be parsed. Aggregators will handle every passed request to yield a meaningfull results.

  • Use the build -function to build a new controller instance.

  • Use the run! method to start the parser and send the requests to the aggregators.

Note that the order of sources can be imported if you have log files than succeed eachother. Requests that span over succeeding files will be parsed correctly if the sources are registered in the correct order. This can be helpful to parse requests from several logrotated log files.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(source, options = {}) ⇒ Controller

Builds a new Controller for the given log file format. format Logfile format. Defaults to :rails Options are passd on to the LogParser.

  • :database Database the controller should use.

  • :yaml Yaml Dump the contrller should use.

  • :output All report outputs get << through this output.

  • :no_progress No progress bar

  • :silent Minimal output, only error



243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
 
# File 'lib/request_log_analyzer/controller.rb', line 243

def initialize(source, options = {})
  @source      = source
  @options     = options
  @aggregators = []
  @filters     = []
  @output      = options[:output]
  @interrupted = false

  # Register the request format for this session after checking its validity
  fail 'Invalid file format!' unless @source.file_format.valid?

  # Install event handlers for wrnings, progress updates and source changes
  @source.warning        = lambda { |type, message, lineno|  @aggregators.each { |agg| agg.warning(type, message, lineno) } }
  @source.progress       = lambda { |message, value| handle_progress(message, value) } unless options[:no_progress]
  @source.source_changes = lambda { |change, filename| handle_source_change(change, filename) }
end

Instance Attribute Details

#aggregators ⇒ Object (readonly)

Returns the value of attribute aggregators.



16
17
18
 
# File 'lib/request_log_analyzer/controller.rb', line 16

def aggregators
  @aggregators
end

#filters ⇒ Object (readonly)

Returns the value of attribute filters.



16
17
18
 
# File 'lib/request_log_analyzer/controller.rb', line 16

def filters
  @filters
end

#options ⇒ Object (readonly)

Returns the value of attribute options.



16
17
18
 
# File 'lib/request_log_analyzer/controller.rb', line 16

def options
  @options
end

#output ⇒ Object (readonly)

Returns the value of attribute output.



16
17
18
 
# File 'lib/request_log_analyzer/controller.rb', line 16

def output
  @output
end

#source ⇒ Object (readonly)

Returns the value of attribute source.



16
17
18
 
# File 'lib/request_log_analyzer/controller.rb', line 16

def source
  @source
end

Class Method Details

.build(options) ⇒ Object

Build a new controller. Returns a new RequestLogAnalyzer::Controller object.

Options

  • :after Drop all requests after this date (Date, DateTime, Time, or a String in “YYYY-MM-DD hh:mm:ss” format)

  • :aggregator Array of aggregators (Strings or Symbols for the builtin aggregators or a RequestLogAnalyzer::Aggregator class - Defaults to [:summarizer]).

  • :boring Do not show color on STDOUT (Defaults to false).

  • :before Drop all requests before this date (Date, DateTime, Time or a String in “YYYY-MM-DD hh:mm:ss” format)

  • :database Database file to insert encountered requests to.

  • :debug Enables echo aggregator which will echo each request analyzed.

  • :file Filestring, File or StringIO.

  • :format :rails, => ‘FORMATSTRING’, :merb, :amazon_s3, :mysql or RequestLogAnalyzer::FileFormat class. (Defaults to :rails).

  • :mail Email the results to this email address.

  • :mailhost Email the results to this mail server.

  • :mailfrom Set the Email sender address.

  • :mailfrom_alias Set the Email sender name.

  • :mailsubject Email subject.

  • :no_progress Do not display the progress bar (increases parsing speed).

  • :output ‘FixedWidth’, ‘HTML’ or RequestLogAnalyzer::Output class. Defaults to ‘FixedWidth’.

  • :reject Reject specific => :value combination (expects a single hash).

  • :report_width Width of reports in characters for FixedWidth reports. (Defaults to 80)

  • :reset_database Reset the database before starting.

  • :select Select specific => :value combination (expects a single hash).

  • :source_files Source files to analyze. Provide either File, array of files or STDIN.

  • :yaml Output to YAML file.

  • :silent Minimal output automatically implies :no_progress

  • :source The class to instantiate to grab the requestes, must be a RequestLogAnalyzer::Source::Base descendant. (Defaults to RequestLogAnalyzer::Source::LogParser)

Example

RequestLogAnalyzer::Controller.build(

:output       => :HTML,
:mail         => 'root@localhost',
:after        => Time.now - 24*60*60,
:source_files => '/var/log/passenger.log'

).run!

Todo

  • Check if defaults work (Aggregator defaults seem wrong).

  • Refactor :database => options, :dump => options away from contoller intialization.



133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
 
# File 'lib/request_log_analyzer/controller.rb', line 133

def self.build(options)
  # Defaults
  options[:output]        ||= 'FixedWidth'
  options[:format]        ||= :rails
  options[:aggregator]    ||= [:summarizer]
  options[:report_width]  ||= 80
  options[:report_amount] ||= 20
  options[:report_sort]   ||= 'sum,mean'
  options[:boring]        ||= false
  options[:silent]        ||= false
  options[:source]        ||= RequestLogAnalyzer::Source::LogParser

  options[:no_progress] = true if options[:silent]

  # Deprecation warnings
  if options[:dump]
    warn '[DEPRECATION] `:dump` is deprecated.  Please use `:yaml` instead.'
    options[:yaml]          = options[:dump]
  end

  # Set the output class
  output_args   = {}
  output_object = nil
  if options[:output].is_a?(Class)
    output_class = options[:output]
  else
    output_class = RequestLogAnalyzer::Output.const_get(options[:output])
  end

  output_sort   = options[:report_sort].split(',').map { |s| s.to_sym }
  output_amount = options[:report_amount] == 'all' ? :all : options[:report_amount].to_i

  if options[:file]
    output_object = %w(        File StringIO        ).include?(options[:file].class.name) ? options[:file] : File.new(options[:file], 'w+')
    output_args   = { width: 80, color: false, characters: :ascii, sort: output_sort, amount: output_amount }
  elsif options[:mail]
    output_object = RequestLogAnalyzer::Mailer.new(options[:mail], options[:mailhost], subject: options[:mailsubject], from: options[:mailfrom], from_alias: options[:mailfrom_name])
    output_args   = { width: 80, color: false, characters: :ascii, sort: output_sort, amount: output_amount  }
  else
    output_object = STDOUT
    output_args   = { width: options[:report_width].to_i, color: !options[:boring],
                    characters: (options[:boring] ? :ascii : :utf), sort: output_sort, amount: output_amount }
  end

  output_instance = output_class.new(output_object, output_args)

  # Create the controller with the correct file format
  if options[:format].is_a?(Hash)
    file_format = RequestLogAnalyzer::FileFormat.load(options[:format].keys[0], options[:format].values[0])
  else
    file_format = RequestLogAnalyzer::FileFormat.load(options[:format])
  end

  # Kickstart the controller
  controller =
    Controller.new(options[:source].new(file_format,
                                        source_files: options[:source_files],
                                        parse_strategy: options[:parse_strategy]),
                   output: output_instance,
                   database: options[:database],                # FUGLY!
                   yaml: options[:yaml],
                   reset_database: options[:reset_database],
                   no_progress: options[:no_progress],
                   silent: options[:silent]
                   )

  # register filters
  if options[:after] || options[:before]
    filter_options = {}
    [:after, :before].each do |filter|
      case options[filter]
      when Date, DateTime, Time
        filter_options[filter] = options[filter]
      when String
        filter_options[filter] = DateTime.parse(options[filter])
      end
    end
    controller.add_filter(:timespan, filter_options)
  end

  if options[:reject]
    options[:reject].each do |(field, value)|
      controller.add_filter(:field, mode: :reject, field: field, value: value)
    end
  end

  if options[:select]
    options[:select].each do |(field, value)|
      controller.add_filter(:field, mode: :select, field: field, value: value)
    end
  end

  # register aggregators
  options[:aggregator].each { |agg| controller.add_aggregator(agg) }
  controller.add_aggregator(:summarizer)          if options[:aggregator].empty?
  controller.add_aggregator(:echo)                if options[:debug]
  controller.add_aggregator(:database_inserter)   if options[:database] && !options[:aggregator].include?('database')

  file_format.setup_environment(controller)
  controller
end

.build_from_arguments(arguments) ⇒ Object

Builds a RequestLogAnalyzer::Controller given parsed command line arguments <tt>arguments<tt> A CommandLine::Arguments hash containing parsed commandline parameters.



20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/request_log_analyzer/controller.rb', line 20

def self.build_from_arguments(arguments)
  options = {}

  # Copy fields
  options[:database]       = arguments[:database]
  options[:reset_database] = arguments[:reset_database]
  options[:debug]          = arguments[:debug]
  options[:yaml]           = arguments[:yaml] || arguments[:dump]
  options[:mail]           = arguments[:mail]
  options[:no_progress]    = arguments[:no_progress]
  options[:format]         = arguments[:format]
  options[:output]         = arguments[:output]
  options[:file]           = arguments[:file]
  options[:after]          = arguments[:after]
  options[:before]         = arguments[:before]
  options[:reject]         = arguments[:reject]
  options[:select]         = arguments[:select]
  options[:boring]         = arguments[:boring]
  options[:aggregator]     = arguments[:aggregator]
  options[:report_width]   = arguments[:report_width]
  options[:report_sort]    = arguments[:report_sort]
  options[:report_amount]  = arguments[:report_amount]
  options[:mailhost]       = arguments[:mailhost]
  options[:mailfrom]       = arguments[:mailfrom]
  options[:mailfrom_name]  = arguments[:mailfrom_name]
  options[:mailsubject]    = arguments[:mailsubject]
  options[:silent]         = arguments[:silent]
  options[:parse_strategy] = arguments[:parse_strategy]

  # Apache format workaround
  if arguments[:rails_format]
    options[:format] = { rails: arguments[:rails_format] }
  elsif arguments[:apache_format]
    options[:format] = { apache: arguments[:apache_format] }
  end

  # Handle output format casing
  if options[:output].class == String
    options[:output] = 'HTML'       if options[:output] =~ /^html$/i
    options[:output] = 'FixedWidth' if options[:output] =~ /^fixed_?width$/i
  end

  # Register sources
  if arguments.parameters.length == 1
    file = arguments.parameters[0]
    if file == '-' || file == 'STDIN'
      options.store(:source_files, $stdin)
    elsif File.exist?(file)
      options.store(:source_files, file)
    else
      puts "File not found: #{file}"
      exit(0)
    end
  else
    options.store(:source_files, arguments.parameters)
  end

  # Guess file format
  if !options[:format] && options[:source_files]
    options[:format] = :rails3 # Default

    if options[:source_files] != $stdin
      if options[:source_files].class == String
        options[:format] = RequestLogAnalyzer::FileFormat.autodetect(options[:source_files])

      elsif options[:source_files].class == Array && options[:source_files].first != $stdin
        options[:format] = RequestLogAnalyzer::FileFormat.autodetect(options[:source_files].first)
      end
    end
  end

  build(options)
end

Instance Method Details

#add_aggregator(agg) ⇒ Object Also known as: >>

Adds an aggregator to the controller. The aggregator will be called for every request that is parsed from the provided sources (see add_source)



288
289
290
291
 
# File 'lib/request_log_analyzer/controller.rb', line 288

def add_aggregator(agg)
  agg = RequestLogAnalyzer::Aggregator.const_get(RequestLogAnalyzer.to_camelcase(agg)) if agg.is_a?(String) || agg.is_a?(Symbol)
  @aggregators << agg.new(@source, @options)
end

#add_filter(filter, filter_options = {}) ⇒ Object

Adds a request filter to the controller.



296
297
298
299
 
# File 'lib/request_log_analyzer/controller.rb', line 296

def add_filter(filter, filter_options = {})
  filter = RequestLogAnalyzer::Filter.const_get(RequestLogAnalyzer.to_camelcase(filter)) if filter.is_a?(Symbol)
  @filters << filter.new(source.file_format, @options.merge(filter_options))
end

#aggregate_request(request) ⇒ Object

Push a request to all the aggregators (@aggregators). request The request to push to the aggregators.



314
315
316
317
318
 
# File 'lib/request_log_analyzer/controller.rb', line 314

def aggregate_request(request)
  return false unless request
  @aggregators.each { |agg| agg.aggregate(request) }
  true
end

#filter_request(request) ⇒ Object

Push a request through the entire filterchain (@filters). request The request to filter. Returns the filtered request or nil.



304
305
306
307
308
309
310
 
# File 'lib/request_log_analyzer/controller.rb', line 304

def filter_request(request)
  @filters.each do |filter|
    request = filter.filter(request)
    return nil if request.nil?
  end
  request
end

#handle_progress(message, value = nil) ⇒ Object

Progress function. Expects :started with file, :progress with current line and :finished or :interrupted when done. message Current state (:started, :finished, :interupted or :progress). value File or current line.



264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
 
# File 'lib/request_log_analyzer/controller.rb', line 264

def handle_progress(message, value = nil)
  case message
  when :started
    @progress_bar = CommandLine::ProgressBar.new(File.basename(value), File.size(value), STDERR)
  when :finished
    @progress_bar.finish
    @progress_bar = nil
  when :interrupted
    if @progress_bar
      @progress_bar.halt
      @progress_bar = nil
    end
  when :progress
    @progress_bar.set(value)
  end
end

#handle_source_change(change, filename) ⇒ Object

Source change handler



282
283
284
 
# File 'lib/request_log_analyzer/controller.rb', line 282

def handle_source_change(change, filename)
  @aggregators.each { |agg| agg.source_change(change, File.expand_path(filename, Dir.pwd)) }
end

#install_signal_handlers ⇒ Object 



361
362
363
364
365
366
367
 
# File 'lib/request_log_analyzer/controller.rb', line 361

def install_signal_handlers
  Signal.trap('INT') do
    handle_progress(:interrupted)
    puts 'Caught interrupt! Stopping parsing...'
    @interrupted = true
  end
end

#run! ⇒ Object

Runs RequestLogAnalyzer

  1. Call prepare on every aggregator

  2. Generate requests from source object

  3. Filter out unwanted requests

  4. Call aggregate for remaning requests on every aggregator

  5. Call finalize on every aggregator

  6. Call report on every aggregator

  7. Finalize Source



328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
 
# File 'lib/request_log_analyzer/controller.rb', line 328

def run!
  # @aggregators.each{|agg| p agg}

  @aggregators.each { |agg| agg.prepare }
  install_signal_handlers

  @source.each_request do |request|
    break if @interrupted
    aggregate_request(filter_request(request))
  end

  @aggregators.each { |agg| agg.finalize }

  @output.header
  @aggregators.each { |agg| agg.report(@output) }
  @output.footer

  @source.finalize

  if @output.io.is_a?(File)
    unless @options[:silent]
      puts
      puts 'Report written to: ' + File.expand_path(@output.io.path)
      puts 'Need an expert to analyze your application?'
      puts 'Mail to [email protected] or visit us at http://railsdoctors.com'
      puts 'Thanks for using request-log-analyzer!'
    end
    @output.io.close
  elsif @output.io.is_a?(RequestLogAnalyzer::Mailer)
    @output.io.mail
  end
end