Cassandra plugin for Fluentd

Cassandra output plugin for Fluentd.

Implemented using the cassandra-cql gem and targets CQL 3.0.0 and Cassandra 1.1.x

Raison d'être

Currently, there's another Fluentd Cassandra plugin

It's implemented via the Twitter Cassandra gem, which:

a) doesn't provide all of the niceties of CQL, i.e., create/alter/delete keyspaces/columnfamilies
b) doesn't allow a desktop client to make a call to a Cassandra instance hosted on EC2
   (the gem resolves a cassandra node's IP address to its private EC2
    IP address (ex: 10.x.x.x), which isn't accessible outside EC2)

Installation

via RubyGems

fluent-gem install fluent-plugin-cassandra-cql

Quick Start

Cassandra Configuration

# create keyspace (via CQL)
  CREATE KEYSPACE \"FluentdLoggers\" WITH strategy_class='org.apache.cassandra.locator.SimpleStrategy' AND strategy_options:replication_factor=1;

# create table (column family)
  CREATE TABLE events (id varchar, ts bigint, payload text, PRIMARY KEY (id, ts)) WITH CLUSTERING ORDER BY (ts DESC);

# NOTE: schema definition should match that specified in the Fluentd.conf configuration file (see below)

Fluentd.conf Configuration

<match cassandra_cql.**>
  type cassandra_cql         # fluent output plugin file name (sans fluent_plugin_ prefix)
  host 127.0.0.1             # cassandra hostname.
  port 9160                  # cassandra thrft port.
  keyspace FluentdLoggers    # cassandra keyspace
  columnfamily spec_events   # cassandra column family
  ttl 60                     # cassandra ttl *optional => default is 0*
  schema                     # cassandra column family schema *hash where keys => column names and values => data types*
  data_keys                  # comma delimited string of the fluentd hash's keys
  pop_data_keys              # keep or pop key/values from the fluentd hash when storing it as json
</match>

Tests

rake rspec

NOTE: requires that cassandra be installed on the machine running the
      test as well as a keyspace named "FluentdLoggers" and a column family
      named "spec_events"

TODOs

1) make host and port configurable for tests
2) add rake task to generate keyspace and columnfamily