fluent-plugin-geoip Build Status

Fluentd Output plugin to add information about geographical location of IP addresses with Maxmind GeoIP databases.

fluent-plugin-geoip has bundled cost-free GeoLite City database by default.
Also you can use purchased GeoIP City database (lang:ja) which costs starting from $50.

The accuracy details for GeoLite City (free) and GeoIP City (purchased) has described at the page below.

Dependency

before use, install dependent library as:

# for RHEL/CentOS
$ sudo yum install geoip-devel --enablerepo=epel

# for Ubuntu/Debian
$ sudo apt-get install libgeoip-dev

Installation

install with gem or fluent-gem command as:

# for fluentd
$ gem install fluent-plugin-geoip

# for td-agent
$ sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-geoip

Usage

<match access.apache>
  type geoip

  # Specify one or more geoip lookup field which has ip address (default: host)
  # in the case of accessing nested value, delimit keys by dot like 'host.ip'.
  geoip_lookup_key  host

  # Specify optional geoip database (using bundled GeoLiteCity databse by default)
  geoip_database    '/path/to/your/GeoIPCity.dat'

  # Set adding field with placeholder (more than one settings are required.)
  <record>
    city            ${city['host']}
    latitude        ${latitude['host']}
    longitude       ${longitude['host']}
    country_code3   ${country_code3['host']}
    country_code2   ${country_code2['host']}
    country         ${country_code['host']}
    country_name    ${country_name['host']}
    dma             ${dma_code['host']}
    area            ${area_code['host']}
    region          ${region['host']}
  </record>

  # Settings for tag
  remove_tag_prefix access.
  tag               geoip.${tag}

  # Set log_level for fluentd-v0.10.43 or earlier (default: warn)
  log_level         info

  # Set buffering time (default: 0s)
  flush_interval    1s
</match>

Tips: how to geolocate multiple key

<match access.apache>
  type geoip
  geoip_lookup_key  user1_host, user2_host
  <record>
    user1_city      ${city['user1_host']}
    user2_city      ${city['user2_host']}
  </record>
  remove_tag_prefix access.
  tag               geoip.${tag}
</match>

Advanced config samples

It is a sample to get friendly geo point recdords for elasticsearch with Yajl (JSON) parser.

<match input.access>
  type                   geoip
  geoip_lookup_key       host
  <record>
    # lat lon as properties
    # ex. {"lat" => 37.4192008972168, "lon" => -122.05740356445312 }
    location_properties  { "lat":${latitude['host']}, "lon":${longitude['host']}}

    # lat lon as string
    # ex. "37.4192008972168,-122.05740356445312"
    location_string      ${latitude['host']},${longitude['host']}

    # lat lon as array (it is useful for Kibana's bettermap.)
    # ex. [-122.05740356445312, 37.4192008972168]
    location_array       [${longitude['host']},${latitude['host']}]
  </record>
  remove_tag_prefix      access.
  tag                    geoip.${tag}
</match>

Tutorial

configuration

<source>
  type forward
</source>

<match test.geoip>
  type copy
  <store>
    type stdout
  </store>
  <store>
    type    geoip
    geoip_lookup_key  host
    <record>
      city  ${city['host']}
      lat   ${latitude['host']}
      lon   ${longitude['host']}
    </record>
    remove_tag_prefix test.
    tag     debug.${tag}
  </store>
</match>

<match debug.**>
  type stdout
</match>

result

# forward record with Google's ip address.
$ echo '{"host":"66.102.9.80","message":"test"}' | fluent-cat test.geoip

# check the result at stdout
$ tail /var/log/td-agent/td-agent.log
2013-08-04 16:21:32 +0900 test.geoip: {"host":"66.102.9.80","message":"test"}
2013-08-04 16:21:32 +0900 debug.geoip: {"host":"66.102.9.80","message":"test","city":"Mountain View","lat":37.4192008972168,"lon":-122.05740356445312}

For more details of geoip data format is described at the page below in section GeoIP City Edition CSV Database Fields.
http://dev.maxmind.com/geoip/legacy/csv/

Placeholders

Provides these placeholders for adding field of geolocate results.

  • $city[lookup_field]
  • $latitude[lookup_field]
  • $longitude[lookup_field]
  • $country_code3[lookup_field]
  • $country_code2[lookup_field]
  • $country_code[lookup_field]
  • $country_name[lookup_field]
  • $dma_code[lookup_field]
  • $area_code[lookup_field]
  • $region[lookup_field]

Parameters

  • include_tag_key (default: false)
  • tag_key

Add original tag name into filtered record using SetTagKeyMixin.
Further details are written at http://docs.fluentd.org/articles/in_exec

  • remove_tag_prefix
  • remove_tag_suffix
  • add_tag_prefix
  • add_tag_suffix

Set one or more option are required unless using tag option for editing tag name. (HandleTagNameMixin feature)

  • tag

On using this option with tag placeholder like tag geoip.${tag} (test code is available at test_out_geoip.rb), it will be overwrite after these options affected. which are remove_tag_prefix, remove_tag_suffix, add_tag_prefix and add_tag_suffix.

  • flush_interval (default: 0 sec)

Set buffering time to execute bulk lookup geoip.

Articles

TODO

Pull requests are very welcome!!

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Copyright (c) 2013- Kentaro Yoshida (@yoshi_ken)

License

Apache License, Version 2.0

This product includes GeoLite data created by MaxMind, available from http://www.maxmind.com.