Class: LogStash::Outputs::ElasticSearch

Inherits:

Base

Object
Base
LogStash::Outputs::ElasticSearch

Includes:: Common, CommonConfigs

Defined in:: lib/logstash/outputs/elasticsearch.rb,
lib/logstash/outputs/elasticsearch/common.rb,
lib/logstash/outputs/elasticsearch/safe_url.rb,
lib/logstash/outputs/elasticsearch/http_client.rb,
lib/logstash/outputs/elasticsearch/common_configs.rb,
lib/logstash/outputs/elasticsearch/http_client/pool.rb,
lib/logstash/outputs/elasticsearch/template_manager.rb,
lib/logstash/outputs/elasticsearch/http_client_builder.rb,
lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb

Overview

This plugin is the recommended method of storing logs in Elasticsearch. If you plan on using the Kibana web interface, you’ll want to use this output.

This output only speaks the HTTP protocol. HTTP is the preferred protocol for interacting with Elasticsearch as of Logstash 2.0. We strongly encourage the use of HTTP over the node protocol for a number of reasons. HTTP is only marginally slower, yet far easier to administer and work with. When using the HTTP protocol one may upgrade Elasticsearch versions without having to upgrade Logstash in lock-step. For those still wishing to use the node or transport protocols please see the <<plugins-outputs-elasticsearch_java,elasticsearch_java output plugin>>.

You can learn more about Elasticsearch at <www.elastic.co/products/elasticsearch>

Template management for Elasticsearch 5.x

Index template for this version (Logstash 5.0) has been changed to reflect Elasticsearch’s mapping changes in version 5.0. Most importantly, the subfield for string multi-fields has changed from ‘.raw` to `.keyword` to match ES default behavior.

** Users installing ES 5.x and LS 5.x ** This change will not affect you and you will continue to use the ES defaults.

** Users upgrading from LS 2.x to LS 5.x with ES 5.x ** LS will not force upgrade the template, if ‘logstash` template already exists. This means you will still use `.raw` for sub-fields coming from 2.x. If you choose to use the new template, you will have to reindex your data after the new template is installed.

Retry Policy

The retry policy has changed significantly in the 2.2.0 release. This plugin uses the Elasticsearch bulk API to optimize its imports into Elasticsearch. These requests may experience either partial or total failures.

The following errors are retried infinitely:

Network errors (inability to connect)
429 (Too many requests) and
503 (Service unavailable) errors

NOTE: 409 exceptions are no longer retried. Please set a higher ‘retry_on_conflict` value if you experience 409 exceptions. It is more performant for Elasticsearch to retry these exceptions than this plugin.

DNS Caching

This plugin uses the JVM to lookup DNS entries and is subject to the value of docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html[networkaddress.cache.ttl], a global setting for the JVM.

As an example, to set your DNS TTL to 1 second you would set the ‘LS_JAVA_OPTS` environment variable to `-Dnetworkaddress.cache.ttl=1`.

Keep in mind that a connection with keepalive enabled will not reevaluate its DNS value while the keepalive is in effect.

Defined Under Namespace

Modules: Common, CommonConfigs, HttpClientBuilder, SafeURL Classes: HttpClient, TemplateManager

Constant Summary collapse

@@plugins =

Gem::Specification.find_all{|spec| spec.name =~ /logstash-output-elasticsearch-/ }

Constants included from Common

Common::RETRYABLE_CODES, Common::SUCCESS_CODES, Common::VALID_HTTP_ACTIONS

Instance Attribute Summary

Attributes included from Common

#client, #hosts

Instance Method Summary collapse

Methods included from Common

#check_action_validity, #event_action_params, #event_action_tuple, #get_event_type, #install_template, #multi_receive, #next_sleep_interval, #register, #retrying_submit, #safe_bulk, #setup_hosts, #sleep_for_interval, #submit, #valid_actions

Methods included from CommonConfigs

included

Instance Method Details

#build_client ⇒ `Object`



187
188
189

# File 'lib/logstash/outputs/elasticsearch.rb', line 187

def build_client
  @client = ::LogStash::Outputs::ElasticSearch::HttpClientBuilder.build(@logger, @hosts, params)
end

#close ⇒ `Object`



191
192
193

# File 'lib/logstash/outputs/elasticsearch.rb', line 191

def close
  @stopping.make_true
end