InfluxDB::AsyncQueue
influxdb-async_queue
is an attempt to optimize influxdb interactions by aggregating metrics into batches through redis queue.
The gem consists of 2 major components:
InflunxDB::AsyncQueue::Queue
responds towrite_point
/write_points
and could be used instead ofInfluxDB::Client
InfluxDB::AsyncQueue::Server
grabs a batch of points from a queue and sends them to influxdb in an infinite loop.
Installation
Add this line to your application's Gemfile:
gem 'influxdb-async_queue'
And then execute:
$ bundle
Or install it yourself as:
$ gem install influxdb-async_queue
Usage
Configuration
Configure manually
InfluxDB::AsyncQueue.configure do |config|
config.influxdb_database = 'test'
config.influxdb_hosts = ['localhost']
config.adapter_name = 'redis'
end
load configuration from file
InfluxDB::AsyncQueue.load_config('path/to/file.yml')
batch_size: 100
sleep_timeout: 0
influxdb:
database: example
username: root
password: root
port: 8086
precision: ms
hosts:
- localhost
adapter:
name: redis
config:
-
host: localhost
port: 6379
db: 0
or combine both options
InfluxDB::AsyncQueue.load_config('path/to/file.yml') do |config|
config.logger = Logger.new
config.influxdb_database = 'test'
end
Writing points
Use InfluxDB::AsyncQueue::Queue
instead of InfluxDB::Client
to push points into queue
client = InfluxDB::AsyncQueue::Client.queue
# or
client = InfluxDB::AsyncQueue::Client::Queue.new(
InfluxDB::AsyncQueue::Adapters::Redis.new('redis://localhost:6379/0', 'redis-keys')
)
client.write_point('series_name', tags: { tag: 1 }, values: { value: 10 })
client.write_points([
{ series: 'series_name', tags: { tag: 1 }, values: { value: 10 } },
{ series: 'series_name', tags: { tag: 2 }, values: { value: 15 } }
])
Points will be serialized into line protocole and pushed into redis queue.
Sending points to influxdb
Start the server in a separate process
InfluxDB::AsyncQueue.load_config('path/to/file.yml')
InfluxDB::AsyncQueue::Server.run
Complete example (rails app)
# bin/influxdb-async_queue
#!/usr/bin/env ruby
# frozen_string_literal: true
env = ENV['RUN_ENV'] || 'development'
puts "env - #{env}"
require 'influxdb/async_queue'
require 'influxdb/async_queue/server'
require 'active_support/logger'
config_path = File.("../config/influxdb-async_queue/#{env}.yml", __dir__)
$stdout.sync = true
InfluxDB::AsyncQueue.configure(config_path) do |c|
c.logger = ActiveSupport::Logger.new($stdout)
end
InfluxDB::AsyncQueue::Server.run
# config/upstart/Procfile.production
influxdb_async_queue: cd /srv/projects/mtsrbt_cms/current && env RUN_ENV=production bundle exec ruby bin/influxdb-async_queue.rb
# config/initializers/metrics.rb
ActiveSupport::Notifications.subscribe "process_action.action_controller" do |_name, start, finish, id, payload|
controller_runtime = ((finish - start)*1000).ceil
view_runtime = (payload[:view_runtime] || 0).ceil
db_runtime = (payload[:db_runtime] || 0).ceil
client = InfluxDB::AsyncQueue.queue
= { method: "#{payload[:controller]}##{payload[:action]}", server: Socket.gethostname }
client.write_point 'rails.controller', values: { value: controller_runtime }, tags:
client.write_point 'rails.view', values: { value: view_runtime }, tags:
client.write_point 'rails.db', values: { value: db_runtime }, tags:
end
Define an adapter for your favorite storage
class InfxluDB::AsyncQueue::Adapters::MyAdapter
def initialize(arg1, arg2)
end
def push(array_of_points)
# ...
end
def pop(limit)
# ...
end
end
configure influxdb-asynq_queue
to use it instead of redis:
InfluxDB::AsyncQueue.configure do |config|
config.adapter = InfluxDB::AsyncQueue::Adapters::MyAdapter.new('foo', 'bar')
end
or
adapter:
name: my_adapter
config: ['foo', 'bar']
Development
After checking out the repo, run bin/setup
to install dependencies. Then, run rake spec
to run the tests. You can also run bin/console
for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install
. To release a new version, update the version number in version.rb
, and then run bundle exec rake release
, which will create a git tag for the version, push git commits and tags, and push the .gem
file to rubygems.org.
Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/mugimaru/influxdb-async_queue. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.
License
The gem is available as open source under the terms of the MIT License.