LogStashLogger
This gem implements a subclass of Ruby's Logger class that logs directly to logstash. It writes to a logstash listener over a UDP (default) or TCP connection, in logstash JSON format. This is an improvement over writing to a file or syslog since logstash can receive the structured data directly.
Features
- Can write directly to logstash over a UDP or TCP/SSL connection.
- Can write to a file, Redis, a unix socket, or stdout.
- Always writes in logstash JSON format.
- Logger can take a string message, a hash, a
LogStash::Event
, or a JSON string as input. - Events are automatically populated with message, timestamp, host, and severity.
- Easily integrates with Rails via configuration.
Installation
Add this line to your application's Gemfile:
gem 'logstash-logger'
And then execute:
$ bundle
Or install it yourself as:
$ gem install logstash-logger
Basic Usage
require 'logstash-logger'
# Defaults to UDP on 0.0.0.0
logger = LogStashLogger.new(port: 5228)
# Specify host and type (UDP or TCP) explicitly
udp_logger = LogStashLogger.new(type: :udp, host: 'localhost', port: 5228)
tcp_logger = LogStashLogger.new(type: :tcp, host: 'localhost', port: 5229)
# Other types of loggers
file_logger = LogStashLogger.new(type: :file, path: 'log/development.log', sync: true)
unix_logger = LogStashLogger.new(type: :unix, path: '/tmp/sock')
redis_logger = LogStashLogger.new(type: :redis)
stdout_logger = LogStashLogger.new(type: :stdout)
# The following messages are written to UDP port 5228:
logger.info 'test'
# {"message":"test","@timestamp":"2014-05-22T09:37:19.204-07:00","@version":"1","severity":"INFO","host":"[hostname]"}
logger.error '{"message": "error"}'
# {"message":"error","@timestamp":"2014-05-22T10:10:55.877-07:00","@version":"1","severity":"ERROR","host":"[hostname]"}
logger.debug message: 'test', foo: 'bar'
# {"message":"test","foo":"bar","@timestamp":"2014-05-22T09:43:24.004-07:00","@version":"1","severity":"DEBUG","host":"[hostname]"}
logger.warn LogStash::Event.new(message: 'test', foo: 'bar')
# {"message":"test","foo":"bar","@timestamp":"2014-05-22T16:44:37.364Z","@version":"1","severity":"WARN","host":"[hostname]"}
# Tagged logging
logger.tagged('foo') { logger.fatal('bar') }
# {"message":"bar","@timestamp":"2014-05-26T20:35:14.685-07:00","@version":"1","severity":"FATAL","host":"[hostname]","tags":["foo"]}
Logstash Configuration
In order for logstash to correctly receive and parse the events, you will need to
configure and run a listener that uses the json_lines
codec. For example, to receive
events over UDP on port 5228:
input {
udp {
host => "0.0.0.0"
port => 5228
codec => json_lines
}
}
See the samples directory for more configuration samples.
SSL
If you are using TCP then there is the option of adding an SSL certificate to the options hash on initialize.
LogStashLogger.new(type: :tcp, port: 5228, ssl_certificate: "/path/to/certificate.crt")
The SSL certificate and key can be generated using
openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout logstash.key -out logstash.crt
You can also enable SSL without a certificate:
LogStashLogger.new(type: :tcp, port: 5228, ssl_enable: true)
The following Logstash configuration is required for SSL:
input {
tcp {
host => "0.0.0.0"
port => 5228
codec => json_lines
ssl_enable => true
ssl_cert => "/path/to/certificate.crt"
ssl_key => "/path/to/key.key"
}
}
Rails Integration
Verified to work with both Rails 3 and 4.
Add the following to your config/environments/production.rb
:
Common Options
# Optional, Rails sets the default to :info
config.log_level = :debug
# Optional, Rails 4 defaults to true in development and false in production
config.autoflush_log = true
UDP
# Optional, defaults to '0.0.0.0'
config.logstash.host = 'localhost'
# Optional, defaults to :udp.
config.logstash.type = :udp
# Required, the port to connect to
config.logstash.port = 5228
TCP
# Optional, defaults to '0.0.0.0'
config.logstash.host = 'localhost'
# Required, the port to connect to
config.logstash.port = 5228
# Required
config.logstash.type = :tcp
# Optional, enables SSL
config.logstash.ssl_enable = true
Unix Socket
# Required
config.logstash.type = :unix
# Required
config.logstash.path = '/tmp/sock'
Redis
# Required
config.logstash.type = :redis
# Optional, will default to the 'logstash' list
config.logstash.list = 'logstash'
# All other options are passed in to the Redis client
# Supported options include host, port, path, password, url
# Example:
# Optional, Redis will default to localhost
config.logstash.host = 'localhost'
# Optional, Redis will default to port 6379
config.logstash.port = 6379
File
# Required
config.logstash.type = :file
# Optional, defaults to Rails log path
config.logstash.path = 'log/production.log'
By default, every Rails log message will be written to logstash in LogStash::Event
JSON format.
For minimal, more-structured logstash events, try one of the following gems:
Currently these gems output a JSON string, which LogStashLogger then parses.
Future versions of these gems could potentially have deeper integration with LogStashLogger (i.e. by writing LogStash::Event
objects).
Ruby Compatibility
Verified to work with:
- MRI Ruby 1.9.3, 2.0+, 2.1+
- JRuby 1.7+
- Rubinius 2.2+
Ruby 1.8.7 is not supported.
What type of logger should I use?
It depends on your specific needs, but most applications should use the default (UDP). Here are the advantages and disadvantages of each type:
- UDP is faster than TCP because it's asynchronous (fire-and-forget). However, this means that log messages could get dropped. This is okay for many applications.
- TCP verifies that every message has been received via two-way communication. It also supports SSL for secure transmission of log messages over a network. This could slow your app down to a crawl if the TCP listener is under heavy load.
- A file is simple to use, but you will have to worry about log rotation and running out of disk space.
- Writing to a Unix socket is faster than writing to a TCP or UDP port, but only works locally.
- Writing to Redis is good for distributed setups that generate tons of logs. However, you will have another moving part and have to worry about Redis running out of memory.
- Writing to stdout is only recommended for debugging purposes.
For a more detailed discussion of UDP vs TCP, I recommend reading this article: UDP vs. TCP
Breaking changes
Version 0.5+
- The
source
event key has been replaced withhost
to better match the latest logstash. - The
(host, port, type)
constructor has been deprecated in favor of an options hash constructor.
Version 0.4+
LogStash::Event
uses the v1 format starting version 1.2+. If you're using the v1, you'll need to install
LogStashLogger version 0.4+. This is not backwards compatible with the old LogStash::Event
v1.1.5, which uses
the v0 format.
Version 0.3+
Earlier versions of this gem (<= 0.2.1) only implemented a TCP connection. Newer versions (>= 0.3) also implement UDP, and use that as the new default. Please be aware if you are using the default constructor and still require TCP, you should add an additional argument:
# Now defaults to UDP instead of TCP
logger = LogStashLogger.new('localhost', 5228)
# Explicitly specify TCP instead of UDP
logger = LogStashLogger.new('localhost', 5228, :tcp)
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request