fluent-plugin-grepcounter Build Status

Fluentd plugin to count the number of matched messages, and emit if exeeds the threshold.

Configuration

Assume inputs from another plugin are as belows:

syslog.host1: {"message":"20.4.01/13T07:02:11.124202 INFO GET /ping" }
syslog.host1: {"message":"20.4.01/13T07:02:13.232645 WARN POST /auth" }
syslog.host1: {"message":"20.4.01/13T07:02:21.542145 WARN GET /favicon.ico" }
syslog.host1: {"message":"20.4.01/13T07:02:43.632145 WARN POST /login" }

An example of grepcounter configuration:

<match syslog.**>
  type grepcounter
  count_interval 60
  input_key message
  regexp WARN
  exclude favicon.ico
  threshold 1
  add_tag_prefix warn.count
</match>

Then, output bocomes as belows (indented):

warn.count.syslog.host1: {
  "count":2,
  "message":["20.4.01/13T07:02:13.232645 WARN POST /auth","20.4.01/13T07:02:43.632145 WARN POST /login"],
  "input_tag":"syslog.host1",
  "input_tag_last":"host1",
}

Output message by joining with a delimiter

As default, the grepcounter plugin outputs matched message as an array as shown above. You may want to output message as a string, then use delimiter option like:

<match syslog.**>
  type grepcounter
  count_interval 60
  input_key message
  regexp WARN
  exclude favicon.ico
  threshold 1
  add_tag_prefix warn.count
  delimiter \n
</match>

Then, output bocomes as belows (indented). You can see the message field is joined with \n.

warn.count.syslog.host1: {
  "count":2,
  "message":"20.4.01/13T07:02:13.232645 WARN POST /auth\n20.4.01/13T07:02:43.632145 WARN POST /login",
  "input_tag":"syslog.host1",
  "input_tag_last":"host1",
}

Parameters

  • count_interval

    The interval time to count in seconds. Default is 60.

  • input_key field_key

    The target field key to grep out. Use with regexp or exclude.

  • regexp regexp

    The filtering regular expression

  • exclude regexp

    The excluding regular expression like grep -v

  • regexp[1-20] field_key regexp (experimental)

    The target field key and the filtering regular expression to grep out. No message is outputted in this case.

  • exclude[1-20] field_key regexp (experimental)

    The target field key and the excluding regular expression like grep -v. No message is outputted in this case.

  • threshold

    The threshold number to emit. Emit if count value >= specified value.

  • greater_equal

    This is same with threshold option. Emit if count value is greater than or equal to (>=) specified value.

  • greater_than

    Emit if count value is greater than (>) specified value.

  • less_than

    Emit if count value is less than (<) specified value.

  • less_equal

    Emit if count value is less than or equal to (<=) specified value.

  • tag

    The output tag. Required for aggregate all.

  • add_tag_prefix

    Add tag prefix for output message

  • remove_tag_prefix

    Remove tag prefix for output message

  • add_tag_suffix

    Add tag suffix for output message

  • remove_tag_suffix

    Remove tag suffix for output message

  • delimiter

    Output matched messages after joined with the specified delimiter.

  • replace_invalid_sequence

    Replace invalid byte sequence in UTF-8 with '?' character if true

  • store_file

    Store internal count data into a file of the given path on shutdown, and load on statring.

ChangeLog

See CHANGELOG.md for details.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Copyright (c) 2013 Naotoshi SEO. See LICENSE for details.