Class: LogStash::Codecs::Multiline
- Defined in:
- lib/logstash/codecs/multiline.rb
Overview
The multiline codec is for taking line-oriented text and merging them into a single event.
The original goal of this codec was to allow joining of multi-line messages from files into a single event. For example - joining java exception and stacktrace messages into a single event.
The config looks like this:
input {
stdin {
codec => multiline {
pattern => "pattern, a regexp"
negate => true or false
what => "previous" or "next"
}
}
}
The ‘pattern’ should match what you believe to be an indicator that the field is part of a multi-line event.
The ‘what’ must be “previous” or “next” and indicates the relation to the multi-line event.
The ‘negate’ can be “true” or “false” (defaults false). If true, a message not matching the pattern will constitute a match of the multiline filter and the what will be applied. (vice-versa is also true)
For example, java stack traces are multiline and usually have the message starting at the far-left, then each subsequent line indented. Do this:
input {
stdin {
codec => multiline {
pattern => "^\s"
what => "previous"
}
}
}
This says that any line starting with whitespace belongs to the previous line.
Another example is to merge lines not starting with a date up to the previous line..
input {
file {
path => "/var/log/someapp.log"
codec => multiline {
# Grok pattern names are valid! :)
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => previous
}
}
}
This is the base class for logstash codecs.
Constant Summary
Constants included from LogStash::Config::Mixin
LogStash::Config::Mixin::CONFIGSORT
Instance Attribute Summary
Attributes included from LogStash::Config::Mixin
Attributes inherited from Plugin
Instance Method Summary collapse
-
#buffer(text) ⇒ Object
def decode.
- #decode(text, &block) ⇒ Object
- #do_next(text, matched, &block) ⇒ Object
- #do_previous(text, matched, &block) ⇒ Object
- #encode(data) ⇒ Object
- #flush(&block) ⇒ Object
- #register ⇒ Object
Methods inherited from Base
#clone, #initialize, #on_event, #teardown
Methods included from LogStash::Config::Mixin
Methods inherited from Plugin
#eql?, #finished, #finished?, #hash, #initialize, #inspect, lookup, #reload, #running?, #shutdown, #teardown, #terminating?, #to_s
Constructor Details
This class inherits a constructor from LogStash::Codecs::Base
Instance Method Details
#buffer(text) ⇒ Object
def decode
157 158 159 160 |
# File 'lib/logstash/codecs/multiline.rb', line 157 def buffer(text) @time = Time.now.utc if @buffer.empty? @buffer << text end |
#decode(text, &block) ⇒ Object
141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 |
# File 'lib/logstash/codecs/multiline.rb', line 141 def decode(text, &block) text.force_encoding(@charset) if @charset != "UTF-8" # Convert to UTF-8 if not in that character set. text = text.encode("UTF-8", :invalid => :replace, :undef => :replace) end match = @grok.match(text) @logger.debug("Multiline", :pattern => @pattern, :text => text, :match => !match.nil?, :negate => @negate) # Add negate option match = (match and !@negate) || (!match and @negate) @handler.call(text, match, &block) end |
#do_next(text, matched, &block) ⇒ Object
173 174 175 176 |
# File 'lib/logstash/codecs/multiline.rb', line 173 def do_next(text, matched, &block) buffer(text) flush(&block) if !matched end |
#do_previous(text, matched, &block) ⇒ Object
178 179 180 181 |
# File 'lib/logstash/codecs/multiline.rb', line 178 def do_previous(text, matched, &block) flush(&block) if !matched buffer(text) end |
#encode(data) ⇒ Object
184 185 186 187 |
# File 'lib/logstash/codecs/multiline.rb', line 184 def encode(data) # Nothing to do. @on_event.call(data) end |
#flush(&block) ⇒ Object
162 163 164 165 166 167 168 169 170 171 |
# File 'lib/logstash/codecs/multiline.rb', line 162 def flush(&block) if @buffer.any? event = LogStash::Event.new("@timestamp" => @time, "message" => @buffer.join("\n")) # Tag multiline events event.tag @multiline_tag if @multiline_tag && @buffer.size > 1 yield event @buffer = [] end end |
#register ⇒ Object
103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 |
# File 'lib/logstash/codecs/multiline.rb', line 103 def register require "grok-pure" # rubygem 'jls-grok' # Detect if we are running from a jarfile, pick the right path. patterns_path = [] if __FILE__ =~ /file:\/.*\.jar!.*/ patterns_path += ["#{File.dirname(__FILE__)}/../../patterns/*"] else patterns_path += ["#{File.dirname(__FILE__)}/../../../patterns/*"] end @grok = Grok.new @patterns_dir = patterns_path.to_a + @patterns_dir @patterns_dir.each do |path| # Can't read relative paths from jars, try to normalize away '../' while path =~ /file:\/.*\.jar!.*\/\.\.\// # replace /foo/bar/../baz => /foo/baz path = path.gsub(/[^\/]+\/\.\.\//, "") end if File.directory?(path) path = File.join(path, "*") end Dir.glob(path).each do |file| @logger.info("Grok loading patterns from file", :path => file) @grok.add_patterns_from_file(file) end end @grok.compile(@pattern) @logger.debug("Registered multiline plugin", :type => @type, :config => @config) @buffer = [] @handler = method("do_#{@what}".to_sym) end |