Elasticshelf

Manage Elasticsearch indices using wither, close, open, delete, snapshot, and restore.

Most of the index actions can be performed by setting an expiry via cutoff days.

The gem may be used within a Rails app or in a Ruby script for cron jobs.

Note: this is inspired by Curator at https://github.com/elasticsearch/curator.git, but if you are an ophidiophobe, no worries, this is written in Ruby :-)

The following actions may be performed on Elasticsearch indices:

wither (i.e. disable bloom filter)
close
open
delete
snapshot
restore

Installation

Add this line to your application's Gemfile:

gem 'elasticshelf'

And then execute:

$ bundle

Or install it yourself as:

$ gem install elasticshelf

Usage

Note: see the examples folder for more command line ruby scripts.

initial setup:

require 'elasticshelf'
es = Elasticshelf::Client.new(:host => '127.0.0.1:9200')

if required be sure to override these defaults:

es.indices_prefix = 'logstash-'
es.date_separator = '.'

Note: an asterisk '*' is appended to indices_prefix.

ensure the cutoff date is as desired:

es.cutoff_days = 30
es.cutoff_days_as_date_to_s

do a "dry run" to see what indices will be affected:

es.find_expired_indices

Wither expired indices (i.e. remove bloom filter)

remove the bloom filter on any indices older than 30 days, but only if they are not closed already:

es.cutoff_days = 30
es.wither_indices

or wither a single index:

es.wither_index("index_name")

Close expired indices

close any indices older than 60 days:

es.cutoff_days = 60
es.close_expired_indices

or close a single index:

es.close_index("index_name")

Open index

just in case one was closed by mistake
es.open_index("index_name")

Delete expired indices

delete any indices older than 60 days:

es.cutoff_days = 60
es.delete_expired_indices

or delete a single index:

es.delete_index("index_name")

Snapshot create repository

Notes:

some sysadmin prep is needed to ensure the location is accessible by Elasticsearch

repo_type defaults to 'fs'

repo_compressed defaults to true

es.repo = 'name_of_the_snapshot_repository'
es.repo_location = '/var/elasticsearch_snapshots' ... or what was prepped by the sysadmin
es.snapshot_create_repository

Snapshot get repository

Note: if es.repo is not set this gets information about all snapshot repositories.

es.snapshot_get_repository

Snapshot delete repository

es.repo = 'name_of_the_snapshot_repository'
es.snapshot_delete_repository

Snapshot expired indices

take a snapshot of all indices older than 22 days:

es.cutoff_days = 22
es.repo = 'name_of_the_snapshot_repository'
es.snapshot_expired_indices

Note: this snapshot is auto-named like "2014-04-02_15:07:55_utc_logstash-_cutoff_22_days", as there may be many indices within this snapshot. Override this as described below.

override snapshot auto-naming behavior:

es.snapshot_expired_indices_name = "more_desirable_name"
es.cutoff_days = 22
es.repo = 'name_of_the_snapshot_repository'
es.snapshot_expired_indices

Snapshot one or more indices

es.indices = "index1" ... single
es.indices = "index1,index2" ... multiples
es.repo = 'name_of_the_snapshot_repository'
es.snapshot = 'name_of_the_snapshot'
es.snapshot_expired_indices

Snapshot get

Note: if es.snapshot is not set this gets information about all snapshots, i.e. to see all snapshots do es.snapshot=nil.

es.snapshot_get

Snapshot delete

es.snapshot = 'name_of_the_snapshot'
es.snapshot_delete

Snapshot restore

es.repo = 'name_of_the_snapshot_repository'
es.snapshot = "name_of_the_snapshot"
- do one of these:
  1. es.indices = "" # restores all indices in the snapshot
  2. es.indices = "index1,index2" # restore multiple indices
  3. es.indices = "index1" # restore a single index
  - note: es.snapshot_get can be used to list indices in a snapshot
es.snapshot_restore
puts "errors=#{es.errors.inspect}"
puts "results=#{es.results.inspect}"

Other usage

es.index_closed?("index_name")
es.get_elasticsearch_version
es.get_lucene_version
es.get_info
es.get_cluster_state("index_name")
es.get_index_state("index_name")
es.version ... of this gem

Notes

both open and closed indices are included when finding expired indices
can not wither a closed index
can not snapshot a closed index
can not restore an open index from a snapshot
deleting all (i.e. '*' or '_all') indices is not allowed
both es.errors and es.results should be checked after any action
exceptions are not re-raised from the elasticsearch-ruby gem, but the exception's class+message are copied into es.errors
the following defaults are set, but may be overridden:
- es.wait_for_completion = true
- es.ignore_unavailable = true
- es.include_global_state = false
- es.date_separator = '.'
- es.indices_prefix = 'logstash-'
- es.repo_type = 'fs'
- es.repo_compressed = true

Actions

get_settings_index name
find_expired_indices
open_index name
index_closed? name
close_index name
close_expired_indices
delete_index name
delete_expired_indices
wither_index name
wither_expired_indices
snapshot_create_repository
snapshot_get_repository
snapshot_delete_repository
snapshot_expired_indices
snapshot_create
snapshot_get
snapshot_delete
snapshot_restore

Contributing

Fork it ( http://github.com//elasticshelf/fork )
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create new Pull Request