Logstash Output OpenSearch

Welcome!
Project Resources
Configuration for Logstash Output Opensearch Plugin
Code of Conduct
License
Copyright

Welcome!

logstash-output-opensearch is a community-driven, open source fork logstash-output-elasticsearch licensed under the Apache v2.0 License. For more information, see opensearch.org.

The logstash-output-opensearch plugin helps to ship events from Logstash to OpenSearch cluster.

Project Resources

Configuration for Logstash Output Opensearch Plugin

To run the Logstash Output Opensearch plugin, add following configuration in your logstash.conf file. Note: For logstash running with OpenSearch 2.12.0 and higher the admin password needs to be a custom strong password supplied during cluster setup.

output {
    opensearch {
        hosts       => ["hostname:port"]
        user        => "admin"
        password    => "<your-admin-password>"
        index       => "logstash-logs-%{+YYYY.MM.dd}"
    }
}

To run the Logstash Output Opensearch plugin using aws_iam authentication, refer to the sample configuration shown below:

output {
   opensearch {
          hosts => ["hostname:port"]
          auth_type => {
              type => 'aws_iam'
              aws_access_key_id => 'ACCESS_KEY'
              aws_secret_access_key => 'SECRET_KEY'
              region => 'us-west-2'
          }
          index  => "logstash-logs-%{+YYYY.MM.dd}"
   }
}

In addition to the existing authentication mechanisms, if we want to add new authentication then we will be adding them in the configuration by using auth_type.

Example Configuration for basic authentication: Note: For logstash running with OpenSearch 2.12.0 and higher the admin password needs to be a custom strong password supplied during cluster setup.

output {
    opensearch {
          hosts  => ["hostname:port"]
          auth_type => {
              type => 'basic'
              user => 'admin'
              password => '<your-admin-password>'
          }
          index => "logstash-logs-%{+YYYY.MM.dd}"
   }
}

To ingest data into a data stream through logstash, we need to create the data stream and specify the name of data stream and the op_type of create in the output configuration. The sample configuration is shown below: Note: For logstash running with OpenSearch 2.12.0 and higher the admin password needs to be a custom strong password supplied during cluster setup.

output {
    opensearch {
          hosts  => ["https://hostname:port"]
          auth_type => {
              type => 'basic'
              user => 'admin'
              password => '<your-admin-password>'
          }
          index => "my-data-stream"
          action => "create"
   }
}

Starting in 2.0.0, the aws sdk version is bumped to v3. In order for all other AWS plugins to work together, please remove pre-installed aws plugins and install logstash-integration-aws plugin as follows. See also https://github.com/logstash-plugins/logstash-mixin-aws/issues/38

# Remove existing logstash aws plugins and install logstash-integration-aws to keep sdk dependency the same
# https://github.com/logstash-plugins/logstash-mixin-aws/issues/38
/usr/share/logstash/bin/logstash-plugin remove logstash-input-s3
/usr/share/logstash/bin/logstash-plugin remove logstash-input-sqs
/usr/share/logstash/bin/logstash-plugin remove logstash-output-s3
/usr/share/logstash/bin/logstash-plugin remove logstash-output-sns
/usr/share/logstash/bin/logstash-plugin remove logstash-output-sqs
/usr/share/logstash/bin/logstash-plugin remove logstash-output-cloudwatch

/usr/share/logstash/bin/logstash-plugin install --version 0.1.0.pre logstash-integration-aws
/usr/share/logstash/bin/logstash-plugin install --version 2.0.0 logstash-output-opensearch

ECS Compatibility

Elastic Common Schema(ECS) compatibility for V8 was added in 1.3.0. For more details on ECS support refer to this documentation.

Code of Conduct

This project has adopted the Amazon Open Source Code of Conduct. For more information see the Code of Conduct FAQ, or contact [email protected] with any additional questions or comments.

License

This project is licensed under the Apache v2.0 License.