Class: LogStash::Outputs::Loggly

Inherits:
Base
  • Object
show all
Defined in:
lib/logstash/outputs/loggly.rb

Overview

Got a loggly account? Use logstash to ship logs to Loggly!

This is most useful so you can use logstash to parse and structure your logs and ship structured, json events to your Loggly account.

To use this, you’ll need to use a Loggly input with type ‘http’ and ‘json logging’ enabled.

Constant Summary collapse

HTTP_SUCCESS =

HTTP constants

"200"
HTTP_FORBIDDEN = 
"403"
HTTP_NOT_FOUND = 
"404"
HTTP_INTERNAL_SERVER_ERROR = 
"500"
HTTP_GATEWAY_TIMEOUT = 
"504"

Instance Method Summary collapse

Instance Method Details

#build_message_bodies(events) ⇒ Object

Concatenates JSON events to build an API call body.

Will yield before going over the body size limit. May yield more than once.

This is also where we check that each message respects the message size, and where we skip those if they don’t.



214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
 
# File 'lib/logstash/outputs/loggly.rb', line 214

def build_message_bodies(events)
  body = ''
  event_count = 0

  events.each do |event|
    encoded_event = format_message(event)
    event_size = encoded_event.bytesize

    if event_size > @max_event_size
      @logger.warn "Skipping event over max event size",
        :event_size => encoded_event.bytesize, :max_event_size => @max_event_size
      @logger.debug "Skipped event", :event => encoded_event
      next
    end

    if body.bytesize + 1 + event_size > @max_payload_size
      @logger.debug "Flushing events to Loggly", count: event_count, bytes: body.bytesize
      yield body
      body = ''
      event_count = 0
    end

    body << "\n" unless body.bytesize.zero?
    body << encoded_event
    event_count += 1
  end

  if event_count > 0
    @logger.debug "Flushing events to Loggly", count: event_count, bytes: body.bytesize
    yield body
  end
end

#format_message(event) ⇒ Object 



169
170
171
 
# File 'lib/logstash/outputs/loggly.rb', line 169

def format_message(event)
  event.to_json
end

#multi_receive(events) ⇒ Object 



126
127
128
 
# File 'lib/logstash/outputs/loggly.rb', line 126

def multi_receive(events)
  send_batch events.collect { |event| prepare_meta(event) }
end

#receive(event) ⇒ Object 



130
131
132
 
# File 'lib/logstash/outputs/loggly.rb', line 130

def receive(event)
  send_batch [prepare_meta(event)]
end

#register ⇒ Object 



121
122
123
 
# File 'lib/logstash/outputs/loggly.rb', line 121

def register
  @logger.debug "Initializing Loggly Output", @config
end

#send_batch(meta_events) ⇒ Object

Takes an array of meta_events or nils. Will split the batch in appropriate sub-batches per key+tag combination (which need to be posted to different URIs).



175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
 
# File 'lib/logstash/outputs/loggly.rb', line 175

def send_batch(meta_events)
  split_batches(meta_events.compact).each_pair do |k, batch|
    key, tag = *k
    if tag.nil?
      url = "#{@proto}://#{@host}/bulk/#{key}"
    else
      url = "#{@proto}://#{@host}/bulk/#{key}/tag/#{tag}"
    end


    build_message_bodies(batch) do |body|
      perform_api_call url, body
    end
  end
end

#split_batches(events) ⇒ Object

Gets all API calls to the same URI together in common batches.

Expects an array of meta_events {key: ‘…’, tag: ‘…’, event: event } Outputs a hash with event batches split out by key+tag combination.

{ [key1, tag1] => [event1, ...],
  [key2, tag1] => [...],
  [key2, tag2] => [...],
  ... }



199
200
201
202
203
204
205
206
 
# File 'lib/logstash/outputs/loggly.rb', line 199

def split_batches(events)
  events.reduce( Hash.new { |h,k| h[k] = [] } ) do |acc, meta_event|
    key = meta_event[:key]
    tag = meta_event[:tag]
    acc[ [key, tag] ] << meta_event[:event]
    acc
  end
end