File: README — Documentation for beaver (1.4.4)

Beaver, chewing through logs to make something useful

Beaver is a light DSL and command line utility for parsing HTTP and Rails production logs back into usable data. It answers questions like:

How many failed logins have there been today? Were they for the same user? From the same IP?
How many 500, 404, etc. errors yesterday? On what pages?
How many Widgets were created yesterday, and with what data?
Did anyone submit a form with the words “kill them all”? Yikes.
Rails 3.2 tagged logging is cool, but what’s a good way to review them?

For a full list of matchers available to “hit”, see the Beaver::Dam class. For a full list of methods available inside a “hit” block, or to members of the “hits” array in a “dam” block, see the Beaver::Request, Beaver::Parsers::Rails, and Beaver::Parsers::HTTP classes.

Installation

[sudo] gem install beaver

Use beaver with a DSL file

hit :failed_logins, :method => :post, :path => '/login', :status => 401

hit :new_widgets, :path => '/widgets', :method => :post, :status => 302 do
  puts "A Widget named #{params[:widget][:name]} was created!"
end

hit :help, :path => %r|^/help| do
  skip! if path == '/help/page_i_want_to_ignore'
  puts "#{ip} looked for help at #{path}"
end

hit :tagged => 'user 1' do
  puts "user 1 was tagged at #{path} - other tags were: #{tags.join(', ')}"
end

hit :errors, :status => (500..503)

dam :failed_logins do
  puts "Failed logins:"
  hits.group_by { |h| h.params[:username] } do |user, fails|
    puts " #{user}"
    fails.each do |hit|
      puts "  from #{hit.ip} at #{hit.time.to_s}"
    end
  end
end

dam :errors do
  puts tablize(' | ') { |hit| [hit.status, hit.path, hit.ip, hit.time] }
end

Run ”beaver” from the command line, passing in your beaver file and some logs:

beaver my_beaver_file.rb /var/www/rails-app/log/production.log*

Use beaver as a library

require 'rubygems'
require 'beaver'

Beaver.stream '/path/to/httpd/access_logs*' do
  hit :failed_logins, :method => :post, :path => '/login', :status => 401
  dam :failed_logins do
    puts "#{hits.size} failed logins!"
  end
end

Or with even less DSL…

beaver = Beaver.new('/path/to/httpd/access_logs*')
beaver.hit :failed_logins, :method => :post, :path => '/login', :status => 401
beaver.dam :failed_logins do
  puts "#{hits.size} failed logins!"
end
beaver.stream

Or just to parse and return the matching requests…

requests = Beaver.dam('/path/to/rails/production.log*', :method => :put)

Or as just a parser (no filtering)…

requests = Beaver.parse('/path/to/rails/production.log*')

Use beaver as part of the *nix toolchain

It’s difficult to grep through a multi-line log format like Rails’ and output each matching multi-line event (though I hear Google is working on a ‘Context-Free Grep’, which may help solve that). Until then, for Rails anyway, beaver is happy to step in.

beaver --path="/widgets" --method=post,put --tagged=tag1,tag2 /var/www/rails-app/log/production.log

Or format the output to a single line:

beaver --controller=widgets /var/www/rails-app/log/production.log --print "%{ip} hit %{action} using %{method}"

Also accepts log content from pipes and stdin. Use it to filter log files:

cat /var/www/rails-app/log/production.log* | beaver --action=edit

beaver --action=edit < /var/www/rails-app/log/production.log.1 

beaver --action=edit --stdin

Or for dead-simple real-time event monitoring:

tail -f /var/log/nginx/access_log | beaver www.rb

See all options with ‘beaver –help’.

Example use with Logwatch

This assumes 1) you’re rotating your Rails logs daily and 2) you’re running logwatch daily.

Check your beaver DSL file into your Rails app, maybe under /script.

Add a logwatch config file for your new service at /etc/logwatch/conf/services/your_app.conf:

Title = "Your Rails App"
LogFile = NONE

In /etc/logwatch/scripts/services/your_app:

beaver /var/www/your_app/script/beaver.rb --yesterday /var/www/your_app/log/production.log{,.1}

Your Rails app should return appropriate HTTP statuses

Rails does a lot of great things for us, but one thing largely up to us are HTTP status codes. For example, your failed logins are probably returning 200 when they should arguably be returning 400 or 401. It’s easy to do, and very useful to Beaver.

render :action => :login, :status => 401

A detailed description of each status code and when to use it can be found at www.w3.org.

Complex Rails tag querying

Beaver supports complex Rails Tagged Logger tag quering, in both DSL and command-line modes.

DSL

# Matches any request tagged with "foo"
hit :tagged, :tagged => 'foo'
# Could also be written as ['foo']

# Matches any request tagged with "foo" AND "bar"
hit :tagged_with_all, :tagged => 'foo, bar'
# Could also be written ['foo', 'bar']

# Matches any request tagged with "foo" OR "bar"
hit :tagged_with_any, :tagged => [['foo'], ['bar']]

# Matches any request tagged with ("foo" AND "bar") OR ("bar" AND "baz") OR "yay"
hit :tagged_and_or_and_or, :tagged => [['foo', 'bar'], ['bar', 'baz'], ['yay']]
# Could also be written ['foo, bar', 'bar, baz', 'yay']

Command-line

# Matches any request tagged with "foo"
beaver --tagged foo production.log

# Matches any request tagged with "foo" AND "bar"
beaver --tagged foo,bar production.log

# Matches any request tagged with "foo" OR "bar"
beaver --tagged foo --tagged bar production.log

# Matches any request tagged with ("foo" AND "bar") OR ("bar" AND "baz") OR "yay"
beaver --tagged foo,bar --tagged bar,baz --tagged yay production.log

License

Licensed under the Apache License