AllSeeingEye

A highly configurable Gem to collect specific data from requests, and view that data in a cute little Sinatra app.

AllSeeingEye writes information about each request it observes into Redis. What kind of information? Why, whatever you want! From the relatively banal (IP address, URI, user agent) to complicated application-specific data (user account, ordered product, money spent), anything can be recorded by AllSeeingEye, stored in Redis, and indexed to be viewed quickly in Sinatra. With pretty Flot-based graphs, even.

It comes with a Rails2 integration by default, but I think with a little work it'd be pretty easy to get it into Rails3.

Background

Our application receives about half a million requests every day. While we have Google Analytics, and it's totally awesome, many of those requests come from our iPhone app, and those requests don't receive the level of tracking we need. We could load reporting software onto a users' iOS device: but, ethical issues aside, why would we want to? Almost anything we'd be interested in tracking requires server interaction anyway, so it's easier and faster to track it all on the server side.

The obvious solution was to mine the Rails logfiles, but we have multiple servers and weaving all the logs was a pain -- and even when we did, some of the data we wanted wasn't exposed. My next thought was to log requests to our database, but we experience enough database traffic as is without writing every request. A MemCached-backed write-through buffer temporarily worked but when the writes happened the database still got very unhappy... and the table got enormous very quickly.

Redis was a good compromise though. We use it for the truly excellent Resque, so it's already part of our stack. It's fast and, while it's persistent, it's not exactly a tragedy if the Redis DB gets hosed, as none of the request data is so important we can't just start collecting it again. Combined with some code to store arbitrary data, we quickly had a winner on our hands.

So, AllSeeingEye was born out of our need to track incoming requests accurately, quickly, and in a way that is easily displayable to our business users.

...but this early version still has a few pain points I'm working out. So some requests are slower than I'd like. But everything will be fast soon enough!

Installation

Get Redis via the amazing Homebrew: brew install redis

Add AllSeeingEye to your project's Gemfile: gem 'all_seeing_eye'

Install it with Bundler: bundle install

Setup

bundle exec ruby script/generate all_seeing_eye

This puts the AllSeeingEye config file at config/all_seeing_eye.yml. The file is annotated with all the information you need to get started, but some quick explanations:

Timeout

AllSeeingEye's data collection is wrapped by a timeout to ensure speediness. I keep this value at 30 milliseconds (0.03), but feel free to change it to suit your application's needs.

Round To Seconds

One data point per request is too many to sensibly view on a graph. This rounds the created_at time of requests to whatever (I choose 10 minutes), so that all requests within 10 minutes are collected and displayed together.

Redis

Environment and then location of the redis server. If you do not include this key, AllSeeingEye tries to look for a redis.yml or a resque.yml file instead to find the server.

Models

You can define a number of different models to collect data with. Each has its own separate namespace, so data won't be shared between them. I only use one right now (I call it 'request'), but if you want more then go for it.

Each model definition should be structured like this:

model_name:
  first_field_name:
    object: object_name
    method: .method_name
  second_field_name:
    object: hash
    method: "['key']"

Model names and field names are completely arbitrary, and I try to humanize them as best as I can.

Object and method are the literal object and the actual method you want executed, with one exception: the object 'controller' is translated into 'self'. The result of this is stored by AllSeeingEye in redis. So, for example, an object could be "request" and the method could be "request_uri". I'm trying to do a little secure sanity-checking here rather than just directly eval'ing a string, but you can still mess things up pretty badly if you choose a bad object and/or bad method... so try not to do that.

As a helpful hint, recording "request_uri" isn't nearly as helpful as you might think. Try to save requests without any params and without a leading slash by defining a method in ApplicationController instead:

protected
def clean_request
  request.request_uri.split('?').first[1..-1]
end

Then, in your all_seeing_eye.yml:

request:
  uri:
    object: controller
    method: .clean_request

Running

When your Rails application starts, you should see each request contain +++ Request watched by AllSeeingEye. That means AllSeeingEye is running successfully. Good job!

Starting the server is simple. Just type all-seeing-eye from the commandline and Sinatra should start automatically.

Using the web interface is fairly straightforward; the only bit that might need some explanation is searching, which is kinda nifty.

Searching

You can use the text field in the upper-right to search. Right now, AllSeeingEye accepts queries in one of two formats:

One time in the past followed by another time in the past, separated by "to". AllSeeingEye uses Chronic for natural language parsing on times, so you can use "yesterday to now", "last friday to yesterday", "last christmas to last easter" and whatever other query formats Chronic supports.
A field and a value, separated by a colon. So "ip:127.0.0.1" will show all results for that IP address, "account:[email protected]" to find an account by a specific email, and so on. This will automatically try to find the correct model that you're searching for as well, which is helpful.

Extras

I included the nginx config we use for AllSeeingEye as a small bonus. It's in examples/all_seeing_eye-nginx.conf.

Contributing to AllSeeingEye

Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
Fork the project
Start a feature/bugfix branch
Commit and push until you are happy with your contribution
Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.