Class: LogStash::Outputs::GoogleCloudStorage

Inherits:
Base
  • Object
show all
Defined in:
lib/logstash/outputs/google_cloud_storage.rb

Overview

Summary: plugin to upload log events to Google Cloud Storage (GCS), rolling files based on the date pattern provided as a configuration setting. Events are written to files locally and, once file is closed, this plugin uploads it to the configured bucket.

For more info on Google Cloud Storage, please go to: cloud.google.com/products/cloud-storage

In order to use this plugin, a Google service account must be used. For more information, please refer to: developers.google.com/storage/docs/authentication#service_accounts

Recommendation: experiment with the settings depending on how much log data you generate, so the uploader can keep up with the generated logs. Using gzip output can be a good option to reduce network traffic when uploading the log files and in terms of storage costs as well.

USAGE: This is an example of logstash config:

output {

google_cloud_storage {
  bucket => "my_bucket"                                     (required)
  key_path => "/path/to/privatekey.p12"                     (required)
  key_password => "notasecret"                              (optional)
  service_account => "[email protected]"   (required)
  temp_directory => "/tmp/logstash-gcs"                     (optional)
  log_file_prefix => "logstash_gcs"                         (optional)
  max_file_size_kbytes => 1024                              (optional)
  output_format => "plain"                                  (optional)
  date_pattern => "%Y-%m-%dT%H:00"                          (optional)
  flush_interval_secs => 2                                  (optional)
  gzip => false                                             (optional)
  uploader_interval_secs => 60                              (optional)
}

}

Improvements TODO list:

  • Support logstash event variables to determine filename.

  • Turn Google API code into a Plugin Mixin (like AwsConfig).

  • There’s no recover method, so if logstash/plugin crashes, files may not

be uploaded to GCS.

  • Allow user to configure file name.

  • Allow parallel uploads for heavier loads (+ connection configuration if

exposed by Ruby API client)

Constant Summary

Constants included from Config::Mixin

Config::Mixin::CONFIGSORT

Instance Attribute Summary

Attributes included from Config::Mixin

#config, #original_params

Attributes inherited from Plugin

#logger, #params

Instance Method Summary collapse

Methods inherited from Base

#handle, #handle_worker, #initialize, #worker_setup, #workers_not_supported

Methods included from Config::Mixin

#config_init, included

Methods inherited from Plugin

#eql?, #finished, #finished?, #hash, #initialize, #inspect, lookup, #reload, #running?, #shutdown, #terminating?, #to_s

Constructor Details

This class inherits a constructor from LogStash::Outputs::Base

Instance Method Details

#receive(event) ⇒ Object 



137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
 
# File 'lib/logstash/outputs/google_cloud_storage.rb', line 137

def receive(event)
  return unless output?(event)

  @logger.debug("GCS: receive method called", :event => event)

  if (@output_format == "json")
    message = event.to_json
  else
    message = event.to_s
  end

  new_base_path = get_base_path()

  # Time to roll file based on the date pattern? Or is it over the size limit?
  if (@current_base_path != new_base_path || (@max_file_size_kbytes > 0 && @temp_file.size >= @max_file_size_kbytes * 1024))
    @logger.debug("GCS: log file will be closed and uploaded",
                  :filename => File.basename(@temp_file.to_path),
                  :size => @temp_file.size.to_s,
                  :max_size => @max_file_size_kbytes.to_s)
    # Close does not guarantee that data is physically written to disk.
    @temp_file.fsync()
    @temp_file.close()
    initialize_next_log()
  end

  @temp_file.write(message)
  @temp_file.write("\n")

  sync_log_file()

  @logger.debug("GCS: event appended to log file",
                :filename => File.basename(@temp_file.to_path))
end

#register ⇒ Object 



114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
 
# File 'lib/logstash/outputs/google_cloud_storage.rb', line 114

def register
  require "fileutils"
  require "thread"

  @logger.debug("GCS: register plugin")

  @upload_queue = Queue.new
  @last_flush_cycle = Time.now
  initialize_temp_directory()
  initialize_current_log()
  initialize_google_client()
  initialize_uploader()

  if @gzip
    @content_type = 'application/gzip'
  else
    @content_type = 'text/plain'
  end
end

#teardown ⇒ Object 



172
173
174
175
176
177
 
# File 'lib/logstash/outputs/google_cloud_storage.rb', line 172

def teardown
  @logger.debug("GCS: teardown method called")

  @temp_file.fsync()
  @temp_file.close()
end