fluent-plugin-stats Build Status

Fluentd plugin to calculate statistics such as sum, max, min, avg.

Configuration

Example 1

Get sum for xxx_count, max for xxx_max, min for xxx_min, avg for xxx_avg

<match foo.**>
  type stats
  interval 5s
  add_tag_prefix stats

  sum .*_count$
  max .*_max$
  min .*_min$
  avg .*_avg$
</match>

Assuming following inputs are coming:

foo.bar: {"4xx_count":1,"5xx_count":2","reqtime_max":12083,"reqtime_min":10,"reqtime_avg":240.46}
foo.bar: {"4xx_count":4,"5xx_count":2","reqtime_max":24831,"reqtime_min":82,"reqtime_avg":300.46}

then output bocomes as belows:

stats.foo.bar: {"4xx_count":5,"5xx_count":4","reqtime_max":24831,"reqtime_min":10,"reqtime_avg":270.46}

Example 2

Get sum, max, min, avg for the same key

<match foo.**>
  type stats
  interval 5s
  add_tag_prefix stats

  sum ^reqtime$
  max ^reqtime$
  min ^reqtime$
  avg ^reqtime$
  sum_suffix _sum
  max_suffix _max
  min_suffix _min
  avg_suffix _avg
</match>

Assuming following inputs are coming:

foo.bar: {"reqtime":1.000}
foo.bar: {"reqtime":2.000}

then output bocomes as belows:

stats.foo.bar: {"reqtime_sum":3.000,"reqtime_max":2.000,"reqtime_min":1.000,"reqtime_avg":1.500}

Parameters

  • sum, min, max, avg

    Target of calculation. Specify input keys by a regular expression

  • sum_keys, min_keys, max_keys, avg_keys

    Target of calculation. Specify input keys by a string separated by , (comma) such as

    sum_keys 4xx_count,5xx_count
    
  • sum_suffix, min_suffix, max_suffix, avg_suffix

    Add a suffix to keys of the output record

  • interval

    The interval to calculate in seconds. Default is 5s.

  • tag

    The output tag name. Required for aggregate all.

  • add_tag_prefix

    Add tag prefix for output message. Default: 'stats'

  • remove_tag_prefix

    Remove tag prefix for output message.

  • add_tag_suffix

    Add tag suffix for output message.

  • remove_tag_suffix

    Remove tag suffix for output message.

  • remove_tag_slice min..max

    Remove tag parts by slice function. FYI: This option behaves like tag.split('.').slice(min..max).

    For example,

    remove_tag_slice 0..-2
    

    changes an input tag foo.bar.host1 to foo.bar.

  • aggregate

    Aggregation unit. One of all, in_tag, out_tag can be specified. Default is in_tag.

    • all calculate stats for all input messages and emit one message in each interval.
    • in_tag calculate stats for each input tag seperately.
    • out_tag calculate stats for for each tag modified by add_tag_prefix, remove_tag_prefix, or remove_tag_slice.
  • store_file

    Store internal data into a file of the given path on shutdown, and load on starting.

  • zero_emit

    Emit 0 on the next interval. This is useful for some software which requires to reset data such as GrowthForecast .

    stats.foo.bar: {"4xx_count":5,"5xx_count":4","reqtime_max":24831,"reqtime_min":10,"reqtime_avg":270.46}
    # after @interval later
    stats.foo.bar: {"4xx_count":0,"5xx_count":0","reqtime_max":0,"reqtime_min":0,"reqtime_avg":0}
    

ChangeLog

See CHANGELOG.md for details.

ToDo

Get the number of denominator to calculate avg from input json field.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Copyright (c) 2013 Naotoshi Seo. See LICENSE for details.