Fluent::Plugin::Rambler

A variant of S3 output plugin for Amazon Redshift.

This plugin works with a command line utility called "rambler":

https://github.com/k24d/rambler

Installation

Add this line to your application's Gemfile:

gem 'fluent-plugin-rambler'

And then execute:

$ bundle

Or install it yourself as:

$ gem install fluent-plugin-rambler

Configuration

RamblerOutput is a subclass of S3Output, and most configuration parameters are inherited from out_s3.

http://docs.fluentd.org/articles/out_s3

There are additional parameters specific to out_rambler:

<match myapp.event1>
  type rambler

  # for fluent-plugin-s3

  aws_key_id YOUR_AWS_KEY_ID
  aws_sec_key YOUR_AWS_SECRET/KEY
  s3_bucket YOUR_S3_BUCKET_NAME
  s3_endpoint s3-ap-northeast-1.amazonaws.com

  # NOTE: don't forget to put '/' after %{time_slice}.
  s3_object_key_format %{path}%{time_slice}/%{hostname}-%{index}.%{file_extension}

  # remote path in the convension of Rambler time-series tables
  path redshift/DATABASE/VIEW_%Y%m/
  buffer_path /var/log/fluent/DATABASE.VIEW

  # the buffer will be flushed with this time slice
  time_slice_format %Y%m%d
  time_slice_wait 10m
  utc

  # The default value corresponds to Redshift's default timestamp: YYYY-MM-DD HH:MI:SS
  #time_format %Y-%m-%d %H:%M:%S

  # for Rambler

  columns %{time}|key1|key2|...
</match>

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request