The Nest Filter plugin allows you to operate on or with nested data. Its modes of operation are

• nest - Take a set of records and place them in a map

• lift - Take a map by key and lift its records up

Example usage (nest)

As an example using JSON notation, to nest keys matching the Wildcard value Key* under a new key NestKey the transformation becomes,

Example (input)

{
  "Key1"     : "Value1",
  "Key2"     : "Value2",
  "OtherKey" : "Value3"
}

Example (output)

{
  "OtherKey" : "Value3"
  "NestKey"  : {
    "Key1"     : "Value1",
    "Key2"     : "Value2",
  }
}

Example usage (lift)

As an example using JSON notation, to lift keys nested under the Nested_under value NestKey* the transformation becomes,

Example (input)

{
  "OtherKey" : "Value3"
  "NestKey"  : {
    "Key1"     : "Value1",
    "Key2"     : "Value2",
  }
}

Example (output)

{
  "Key1"     : "Value1",
  "Key2"     : "Value2",
  "OtherKey" : "Value3"
}

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the Memory Usage Input Plugin, which outputs the following (example),

[0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]

Example #1 - nest

Command Line

Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.

The following command will load the mem plugin. Then the nest filter will match the wildcard rule to the keys and nest the keys matching Mem.* under the new key NEST.

$ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout

Configuration File

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard Mem.*
    Nest_under Memstats
    Remove_prefix Mem.

Result

The output of both the command line and configuration invocations should be identical and result in the following output.

[2018/04/06 01:35:13] [ info] [engine] started
[0] mem.local: [1522978514.007359767, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Memstats"=>{"total"=>4050908, "used"=>714984, "free"=>3335924}}]

Example #1 - nest and lift undo

This example nests all Mem.* and Swap,* items under the Stats key and then reverses these actions with a lift operation. The output appears unchanged.

Configuration File

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard Mem.*
    Wildcard Swap.*
    Nest_under Stats
    Add_prefix NESTED

[FILTER]
    Name nest
    Match *
    Operation lift
    Nested_under Stats
    Remove_prefix NESTED

Result

[2018/06/21 17:42:37] [ info] [engine] started (pid=17285)
[0] mem.local: [1529566958.000940636, {"Mem.total"=>8053656, "Mem.used"=>6940380, "Mem.free"=>1113276, "Swap.total"=>16532988, "Swap.used"=>1286772, "Swap.free"=>15246216}]

Example #2 - nest 3 levels deep

This example takes the keys starting with Mem.* and nests them under LAYER1, which itself is then nested under LAYER2, which is nested under LAYER3.

Configuration File

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard Mem.*
    Nest_under LAYER1

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard LAYER1*
    Nest_under LAYER2

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard LAYER2*
    Nest_under LAYER3

Result

[0] mem.local: [1524795923.009867831, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "LAYER3"=>{"LAYER2"=>{"LAYER1"=>{"Mem.total"=>4050908, "Mem.used"=>1112036, "Mem.free"=>2938872}}}}]


{
  "Swap.total"=>1046524,
  "Swap.used"=>0,
  "Swap.free"=>1046524,
  "LAYER3"=>{
    "LAYER2"=>{
      "LAYER1"=>{
        "Mem.total"=>4050908,
        "Mem.used"=>1112036,
        "Mem.free"=>2938872
      }
    }
  }
}

Example #3 - multiple nest and lift filters with prefix

This example starts with the 3-level deep nesting of Example 2 and applies the lift filter three times to reverse the operations. The end result is that all records are at the top level, without nesting, again. One prefix is added for each level that is lifted.

Configuration file

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard Mem.*
    Nest_under LAYER1

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard LAYER1*
    Nest_under LAYER2

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard LAYER2*
    Nest_under LAYER3

[FILTER]
    Name nest
    Match *
    Operation lift
    Nested_under LAYER3
    Add_prefix Lifted3_

[FILTER]
    Name nest
    Match *
    Operation lift
    Nested_under Lifted3_LAYER2
    Add_prefix Lifted3_Lifted2_

[FILTER]
    Name nest
    Match *
    Operation lift
    Nested_under Lifted3_Lifted2_LAYER1
    Add_prefix Lifted3_Lifted2_Lifted1_

Result

[0] mem.local: [1524862951.013414798, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996}]


{
  "Swap.total"=>1046524, 
  "Swap.used"=>0, 
  "Swap.free"=>1046524, 
  "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, 
  "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, 
  "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996
}

The Grep Filter plugin allows to match or exclude specific records based in regular expression patterns.

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

In order to start filtering records, you can run the filter from the command line or through the configuration file. The following example assumes that you have a file called lines.txt with the following content

aaa
aab
bbb
ccc
ddd
eee
fff
ggg

Command Line

Note: using the command line mode need special attention to quote the regular expressions properly. It's suggested to use a configuration file.

The following command will load the tail plugin and read the content of lines.txt file. Then the grep filter will apply a regular expression rule over the log field (created by tail plugin) and only pass the records which field value starts with aa:

$ bin/fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o stdout

Configuration File

[INPUT]
    Name   tail
    Path   lines.txt

[FILTER]
    Name   grep
    Match  *
    Regex  log aa

[OUTPUT]
    Name   stdout
    Match  *

The filter allows to use multiple rules which are applied in order, you can have many Regex and Exclude entries as required.

Lua Filter allows you to modify the incoming records using custom Lua Scripts.

Due to the necessity to have a flexible filtering mechanism, now is possible to extend Fluent Bit capabilities writing simple filters using Lua programming language. A Lua based filter takes two steps:

• Configure the Filter in the main configuration

• Prepare a Lua script that will be used by the Filter

Content:

• Configuration Parameters

• Getting Started

• Lua Script Filter API

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

In order to test the filter, you can run the plugin from the command line or through the configuration file. The following examples uses the dummy input plugin for data ingestion, invoke Lua filter using the test.lua script and calls the cb_print() function which only print the same information to the standard output:

Command Line

From the command line you can use the following options:

$ fluent-bit -i dummy -F lua -p script=test.lua -p call=cb_print -m '*' -o null

Configuration File

In your main configuration file append the following Input, Filter & Output sections:

[INPUT]
    Name   dummy

[FILTER]
    Name    lua
    Match   *
    script  test.lua
    call    cb_print

[OUTPUT]
    Name   null
    Match  *

Lua Script Filter API

The life cycle of a filter have the following steps:

• Upon Tag matching by filter_lua, it may process or bypass the record.

• If filter_lua accepts the record, it will invoke the function defined in the call property which basically is the name of a function defined in the Lua script.

• Invoke Lua function passing each record in JSON format.

• Upon return, validate return value and take some action (described above)

Callback Prototype

The Lua script can have one or multiple callbacks that can be used by filter_lua, it prototype is as follows:

function cb_print(tag, timestamp, record)
   return code, timestamp, record
end

Function Arguments

Return Values

Each callback must return three values:

Code Examples

For functional examples of this interface, please refer to the code samples provided in the source code of the project located here:

https://github.com/fluent/fluent-bit/tree/master/scripts

Number Type

In Lua, Fluent Bit treats number as double. It means an integer field (e.g. IDs, log levels) will be converted double. To avoid type conversion, Type_int_key property is available.

The Record Modifier Filter plugin allows to append fields or to exclude specific fields.

Configuration Parameters

The plugin supports the following configuration parameters: Remove_key and Whitelist_key are exclusive.

Getting Started

In order to start filtering records, you can run the filter from the command line or through the configuration file.

This is a sample in_mem record to filter.

{"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724}

Append fields

The following configuration file is to append product name and hostname (via environment variable) to record.

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name record_modifier
    Match *
    Record hostname ${HOSTNAME}
    Record product Awesome_Tool

You can also run the filter from command line.

$ fluent-bit -i mem -o stdout -F record_modifier -p 'Record=hostname ${HOSTNAME}' -p 'Record=product Awesome_Tool' -m '*'

The output will be

[0] mem.local: [1492436882.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724, "hostname"=>"localhost.localdomain", "product"=>"Awesome_Tool"}]

Remove fields with Remove_key

The following configuration file is to remove 'Swap.*' fields.

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name record_modifier
    Match *
    Remove_key Swap.total
    Remove_key Swap.used
    Remove_key Swap.free

You can also run the filter from command line.

$ fluent-bit -i mem -o stdout -F  record_modifier -p 'Remove_key=Swap.total' -p 'Remove_key=Swap.free' -p 'Remove_key=Swap.used' -m '*'

The output will be

[0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]

Remove fields with Whitelist_key

The following configuration file is to remain 'Mem.*' fields.

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name record_modifier
    Match *
    Whitelist_key Mem.total
    Whitelist_key Mem.used
    Whitelist_key Mem.free

You can also run the filter from command line.

$ fluent-bit -i mem -o stdout -F  record_modifier -p 'Whitelist_key=Mem.total' -p 'Whitelist_key=Mem.free' -p 'Whitelist_key=Mem.used' -m '*'

The output will be

[0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]

The filter plugins allows to alter the incoming data generated by the input plugins. As of this version the following filter plugins are available:

In order to let a Filter be applied over some data, the Match rule must exists and it must match the Tag for the incoming data.

The Modify Filter plugin allows you to change records using rules and conditions.

Example usage

As an example using JSON notation to,

• Rename Key2 to RenamedKey

• Add a key OtherKey with value Value3 if OtherKey does not yet exist

Example (input)

Example (output)

Configuration Parameters

Rules

The plugin supports the following rules:

• Rules are case insensitive, parameters are not

• Any number of rules can be set in a filter instance.

• Rules are applied in the order they appear, with each rule operating on the result of the previous rule.

Conditions

The plugin supports the following conditions:

• Conditions are case insensitive, parameters are not

• Any number of conditions can be set.

• Conditions apply to the whole filter instance and all its rules. Not to individual rules.

• All conditions have to be true for the rules to be applied.

Example #1 - Add and Rename

Using command Line

Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.

Configuration File

Result

The output of both the command line and configuration invocations should be identical and result in the following output.

Example #2 - Conditionally Add and Remove

Configuration File

Result

Example #3 - Emoji

Configuration File

Result

The Parser Filter plugin allows to parse field in event records.

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

Configuration File

This is an example to parser a record {"data":"100 0.5 true This is example"}.

The plugin needs parser file which defines how to parse field.

The path of parser file should be written in configuration file at [SERVICE] section.

The output is

You can see the record {"data":"100 0.5 true This is example"} are parsed.

Preserve original fields

By default, the parser plugin only keeps the parsed fields in its output.

If you enable Preserve_Key, the original key field is preserved:

This will produce the output:

If you enable Reserve_Data, all other fields are preserved:

This will produce the output:

The Standard Output Filter plugin allows to print to the standard output the data received through the input plugin.

Configuration Parameters

There are no parameters.

Getting Started

In order to start filtering records, you can run the filter from the command line or through the configuration file.

Command Line

$ fluent-bit -i cpu -t cpu.local -F stdout -m '*' -o null -m '*'

Configuration File

In your main configuration file append the following FILTER sections:

[INPUT]
    Name cpu
    Tag  cpu.local

[FILTER]
    Name  stdout
    Match *

[OUTPUT]
    Name  null
    Match *

The Throttle Filter plugin sets the average Rate of messages per Interval, based on leaky bucket and sliding window algorithm. In case of overflood, it will leak within certain rate.

Configuration Parameters

The plugin supports the following configuration parameters:

Functional description

Lets imagine we have configured:

Rate 5
Window 5
Interval 1s

we received 1 message first second, 3 messages 2nd, and 5 3rd. As you can see, disregard that Window is actually 5, we use "slow" start to prevent overflooding during the startup.

+-------+-+-+-+ 
|1|3|5| | | | | 
+-------+-+-+-+ 
|  3  |         average = 3, and not 1.8 if you calculate 0 for last 2 panes. 
+-----+

But as soon as we reached Window size * Interval, we will have true sliding window with aggregation over complete window.

+-------------+ 
|1|3|5|7|3|4| | 
+-------------+ 
  |  4.4    |   
  ----------+

When we have average over window is more than Rate, we will start dropping messages, so that

+-------------+
|1|3|5|7|3|4|7|
+-------------+
    |   5.2   |
    +---------+

will become:

+-------------+
|1|3|5|7|3|4|6|
+-------------+
    |   5     |
    +---------+

As you can see, last pane of the window was overwritten and 1 message was dropped.

Interval vs Window size

You might noticed possibility to configure Interval of the Window shift. It is counter intuitive, but there is a difference between two examples above:

Rate 60
Window 5
Interval 1m

and

Rate 1
Window 300
Interval 1s

Even though both examples will allow maximum Rate of 60 messages per minute, first example may get all 60 messages within first second, and will drop all the rest for the entire minute:

XX        XX        XX
XX        XX        XX
XX        XX        XX
XX        XX        XX
XX        XX        XX
XX        XX        XX
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

While the second example will not allow more than 1 message per second every second, making output rate more smooth:

  X    X     X    X    X    X
XXXX XXXX  XXXX XXXX XXXX XXXX
+-+-+-+-+-+--+-+-+-+-+-+-+-+-+-+

It may drop some data if the rate is ragged. I would recommend to use bigger interval and rate for streams of rare but important events, while keep Window bigger and Interval small for constantly intensive inputs.

Command Line

Note: It's suggested to use a configuration file.

The following command will load the tail plugin and read the content of lines.txt file. Then the throttle filter will apply a rate limit and only pass the records which are read below the certain rate:

$ bin/fluent-bit -i tail -p 'path=lines.txt' -F throttle -p 'rate=1' -m '*' -o stdout

Configuration File

[INPUT]
    Name   tail
    Path   lines.txt

[FILTER]
    Name     throttle
    Match    *
    Rate     1000
    Window   300
    Interval 1s

[OUTPUT]
    Name   stdout
    Match  *

The example above will pass 1000 messages per second in average over 300 seconds.

The Kubernetes Filter allows to enrich your log files with Kubernetes metadata.

When Fluent Bit is deployed in Kubernetes as a DaemonSet and configured to read the log files from the containers (using tail plugin), this filter aims to perform the following operations:

• Analyze the Tag and extract the following metadata:

  • POD Name

  • Namespace

  • Container Name

  • Container ID

• Query Kubernetes API Server to obtain extra metadata for the POD in question:

  • POD ID

  • Labels

  • Annotations

The data is cached locally in memory and appended to each record.

Configuration Parameters

The plugin supports the following configuration parameters:

Kubernetes Annotations

A flexible feature of Fluent Bit Kubernetes filter is that allow Kubernetes Pods to suggest certain behaviors for the log processor pipeline when processing the records. At the moment it support:

• Suggest a pre-defined parser

• Request to exclude logs

The following annotations are available:

Annotation Examples in Pod definition

Suggest a parser

The following Pod definition runs a Pod that emits Apache logs to the standard output, in the Annotations it suggest that the data should be processed using the pre-defined parser called apache:

apiVersion: v1
kind: Pod
metadata:
  name: apache-logs
  labels:
    app: apache-logs
  annotations:
    fluentbit.io/parser: apache
spec:
  containers:
  - name: apache
    image: edsiper/apache_logs

Request to exclude logs

There are certain situations where the user would like to request that the log processor simply skip the logs from the Pod in question:

apiVersion: v1
kind: Pod
metadata:
  name: apache-logs
  labels:
    app: apache-logs
  annotations:
    fluentbit.io/exclude: "true"
spec:
  containers:
  - name: apache
    image: edsiper/apache_logs

Note that the annotation value is boolean which can take a true or false and must be quoted.