Glass Octopus

A Kafka consumer framework. Like Rack but for Kafka.


Add this line to your application's Gemfile:

gem 'glass_octopus'

And then execute:

$ bundle

Or install it yourself as:

$ gem install glass_octopus

This gem requires Ruby 2.1 or higher.

Getting started

Pick your adapter:

  • For Kafka 0.8.x use poseidon and poseidon-cluster

    # in your Gemfile
    gem "glass_octopus"
    gem "poseidon", github: "bpot/poseidon"
    gem "poseidon_cluster", github: "bsm/poseidon_cluster"
  • For Kafka 0.9+ use ruby-kafka

    # in your Gemfile
    gem "glass_octopus"
    gem "ruby-kafka"
# in app.rb
require "bundler/setup"
require "glass_octopus"

app = do
  use GlassOctopus::Middleware::CommonLogger

  run { |ctx|
    puts "Got message: #{ctx.message.key} => #{ctx.message.value}"
end do |config|
  config.adapter :ruby_kafka do |kafka|
    kafka.broker_list = %[localhost:9092]
    kafka.topic       = "mytopic"       = "mygroup"
    kafka.client      = { logger: config.logger }

Run it with bundle exec ruby app.rb

Handling Avro messages with Schema Registry

Glass Octopus can be used with Avro messages validated against a schema. For this, you need a running Schema Registry service.
You also need to have the avro_turf gem installed.

# in your Gemfile
gem "avro_turf"

Add the AvroParser middleware with the Schema Registry URL to your app.

# in app.rb
app = do
  use GlassOctopus::Middleware::AvroParser, "http://schema_registry_url:8081"

For more examples look into the example directory.

For the API documentation please see the documentation site


Install docker and docker-compose to run Kafka and zookeeper for tests.

  1. Set the ADVERTISED_HOST environment variable
  2. Run rake docker:up
  3. Now you can run the tests.

Run all tests including integration tests:

$ rake test:all

Running tests without integration tests:

$ rake # or rake test

When you are done run rake docker:down to clean up docker containers.


The gem is available as open source under the terms of the MIT License.