Netflow plugin for Fluentd

Overview

Fluentd input plugin that acts as Netflow v5/v9 collector.

Requirements

fluent-plugin-netflow	fluentd	ruby
>= 1.0.0	>= v0.14.0	>= 2.1
< 1.0.0	>= v0.12.0	>= 1.9

Installation

Use RubyGems:

fluent-gem install fluent-plugin-netflow

Configuration

<source>
  type netflow
  tag netflow.event

  # optional parameters
  bind 192.168.0.1
  port 2055
  cache_ttl 6000
  versions [5, 9]
  definitions /path/to/custom_fields.yaml
</source>

bind

IP address on which the plugin will accept Netflow.
(Default: '0.0.0.0')

port

UDP port number on which tpe plugin will accept Netflow.
(Default: 5140)

cache_ttl

Template cache TTL for Netflow v9 in seconds. Templates not refreshed from the Netflow v9 exporter within the TTL are expired at the plugin.
(Default: 4000)

versions

Netflow versions which are acceptable.
(Default:[5, 9])

switched_times_from_uptime

When set to true, the plugin stores system uptime for first_switched and last_switched instead of ISO8601-formatted absolute time.
(Defaults: false)

definitions

YAML file containing Netflow field definitions to overfide pre-defined templates. Example is like below

---
option:
  4:          # field value
  - :uint8    # field length
  - :protocol # field type

Performance Evaluation

Benchmark for v5 protocol on Macbook Air (Early 2014, 1.7 GHz Intel Core i7):

• 0 packets dropped in 32,000 records/second (for 3,000,000 packets)
• 45,000 records/second in maximum (for flooding netflow packets)

Tested with the packet generator below:

• https://github.com/mshindo/NetFlow-Generator
• ./flowgen -n3000000 -i50 -w1 -p5140 localhost

And configuration:

<source>
  @type  netflow
  tag netflow.event
  bind 0.0.0.0
  port 5140
  switched_times_from_uptime yes
</source>
<match netflow.event>
  @type flowcounter
  unit minute
  count_keys count # missing column for counting events only
  tag flowcount
</match>
<match flowcount>
  @type stdout
</match>

Tips

Use netflow parser in other plugins

require 'fluent/plugin/parser_netflow'

parser = Fluent::Plugin::NetflowParser.new
parser.configure(conf)

# Netflow v5
parser.call(payload) do |time, record|
  # do something
end

# Netflow v9
parser.call(payload, source_ip_address) do |time, record|
  # do something
end

NOTE: If the plugin receives Netflow v9 from multiple sources, provide source_ip_address argument to parse correctly.

Field definition for Netflow v9

Both option and scope fields for Netflow v9 are defined in YAML where two parameters are described for each field value like:

option:
  ...
  4:           # field value
  - :uint8     # field length
  - :protocol  # field type

See RFC3954 document for more details.

When int value specified for field length, the template parser in this plugin will prefer a field length in received template flowset over YAML. The int value in YAML will be used as a default value only when the length in received flowset is invalid.

option:
  1:
  - 4          # means :unit32, which is just a default
  - :in_bytes

When :skip is described for a field, the template parser will learn the length from received template flowset and skip the field when data flowsets are processed.

option:
  ...
  43:
  - :skip

NOTE: The definitions don't exactly reflect RFC3954 in order to cover some illegal implementations which export Netflow v9 in bad field length.

   31:
   - 3  # Some system exports in 4 bytes despite of RFC
   - :ipv6_flow_label
   ...
   48:
   - 1  # Some system exports in 2 bytes despite of RFC
   - :flow_sampler_id

PaloAlto Netflow

PaloAlto Netflow has different field definitionas: See this definitions for PaloAlto Netflow: https://github.com/repeatedly/fluent-plugin-netflow/issues/27#issuecomment-269197495

More speed ?

:bullettrain_side: Try switched_times_from_uptime true option !

TODO

• Netflow v9 protocol parser optimization
• Use Fluentd feature instead of own handlers