Nest - Fluent Bit: Official Manual

The Nest Filter plugin allows you to operate on or with nested data. Its modes of operation are
nest - Take a set of records and place them in a map
lift - Take a map by key and lift its records up
Example usage (nest)
As an example using JSON notation, to nest keys matching the Wildcard value Key* under a new key NestKey the transformation becomes,
Example (input)
1
{
2
  "Key1"     : "Value1",
3
  "Key2"     : "Value2",
4
  "OtherKey" : "Value3"
5
}
Copied!
Example (output)
1
{
2
  "OtherKey" : "Value3"
3
  "NestKey"  : {
4
    "Key1"     : "Value1",
5
    "Key2"     : "Value2",
6
  }
7
}
Copied!
Example usage (lift)
As an example using JSON notation, to lift keys nested under the Nested_under value NestKey* the transformation becomes,
Example (input)
1
{
2
  "OtherKey" : "Value3"
3
  "NestKey"  : {
4
    "Key1"     : "Value1",
5
    "Key2"     : "Value2",
6
  }
7
}
Copied!
Example (output)
1
{
2
  "Key1"     : "Value1",
3
  "Key2"     : "Value2",
4
  "OtherKey" : "Value3"
5
}
Copied!
Configuration Parameters
The plugin supports the following configuration parameters:
Key
Value Format
Operation
Description
Operation
ENUM [nest or lift]
​
Select the operation nest or lift
Wildcard
FIELD WILDCARD
nest
Nest records which field matches the wildcard
Nest_under
FIELD STRING
nest
Nest records matching the Wildcard under this key
Nested_under
FIELD STRING
lift
Lift records nested under the Nested_under key
Add_prefix
FIELD STRING
ANY
Prefix affected keys with this string
Remove_prefix
FIELD STRING
ANY
Remove prefix from affected keys if it matches this string
Getting Started
In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the Memory Usage Input Plugin, which outputs the following (example),
1
[0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
Copied!
Example #1 - nest
Command Line
Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.
The following command will load the mem plugin. Then the nest filter will match the wildcard rule to the keys and nest the keys matching Mem.* under the new key NEST.
1
$ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout
Copied!
Configuration File
1
[INPUT]
2
    Name mem
3
    Tag  mem.local
4
​
5
[OUTPUT]
6
    Name  stdout
7
    Match *
8
​
9
[FILTER]
10
    Name nest
11
    Match *
12
    Operation nest
13
    Wildcard Mem.*
14
    Nest_under Memstats
15
    Remove_prefix Mem.
Copied!
Result
The output of both the command line and configuration invocations should be identical and result in the following output.
1
[2018/04/06 01:35:13] [ info] [engine] started
2
[0] mem.local: [1522978514.007359767, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Memstats"=>{"total"=>4050908, "used"=>714984, "free"=>3335924}}]
Copied!
Example #1 - nest and lift undo
This example nests all Mem.* and Swap,* items under the Stats key and then reverses these actions with a lift operation. The output appears unchanged.
Configuration File
1
[INPUT]
2
    Name mem
3
    Tag  mem.local
4
​
5
[OUTPUT]
6
    Name  stdout
7
    Match *
8
​
9
[FILTER]
10
    Name nest
11
    Match *
12
    Operation nest
13
    Wildcard Mem.*
14
    Wildcard Swap.*
15
    Nest_under Stats
16
    Add_prefix NESTED
17
​
18
[FILTER]
19
    Name nest
20
    Match *
21
    Operation lift
22
    Nested_under Stats
23
    Remove_prefix NESTED
Copied!
Result
1
[2018/06/21 17:42:37] [ info] [engine] started (pid=17285)
2
[0] mem.local: [1529566958.000940636, {"Mem.total"=>8053656, "Mem.used"=>6940380, "Mem.free"=>1113276, "Swap.total"=>16532988, "Swap.used"=>1286772, "Swap.free"=>15246216}]
Copied!
Example #2 - nest 3 levels deep
This example takes the keys starting with Mem.* and nests them under LAYER1, which itself is then nested under LAYER2, which is nested under LAYER3.
Configuration File
1
[INPUT]
2
    Name mem
3
    Tag  mem.local
4
​
5
[OUTPUT]
6
    Name  stdout
7
    Match *
8
​
9
[FILTER]
10
    Name nest
11
    Match *
12
    Operation nest
13
    Wildcard Mem.*
14
    Nest_under LAYER1
15
​
16
[FILTER]
17
    Name nest
18
    Match *
19
    Operation nest
20
    Wildcard LAYER1*
21
    Nest_under LAYER2
22
​
23
[FILTER]
24
    Name nest
25
    Match *
26
    Operation nest
27
    Wildcard LAYER2*
28
    Nest_under LAYER3
Copied!
Result
1
[0] mem.local: [1524795923.009867831, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "LAYER3"=>{"LAYER2"=>{"LAYER1"=>{"Mem.total"=>4050908, "Mem.used"=>1112036, "Mem.free"=>2938872}}}}]
2
​
3
​
4
{
5
  "Swap.total"=>1046524,
6
  "Swap.used"=>0,
7
  "Swap.free"=>1046524,
8
  "LAYER3"=>{
9
    "LAYER2"=>{
10
      "LAYER1"=>{
11
        "Mem.total"=>4050908,
12
        "Mem.used"=>1112036,
13
        "Mem.free"=>2938872
14
      }
15
    }
16
  }
17
}
Copied!
Example #3 - multiple nest and lift filters with prefix
This example starts with the 3-level deep nesting of Example 2 and applies the lift filter three times to reverse the operations. The end result is that all records are at the top level, without nesting, again. One prefix is added for each level that is lifted.
Configuration file
1
[INPUT]
2
    Name mem
3
    Tag  mem.local
4
​
5
[OUTPUT]
6
    Name  stdout
7
    Match *
8
​
9
[FILTER]
10
    Name nest
11
    Match *
12
    Operation nest
13
    Wildcard Mem.*
14
    Nest_under LAYER1
15
​
16
[FILTER]
17
    Name nest
18
    Match *
19
    Operation nest
20
    Wildcard LAYER1*
21
    Nest_under LAYER2
22
​
23
[FILTER]
24
    Name nest
25
    Match *
26
    Operation nest
27
    Wildcard LAYER2*
28
    Nest_under LAYER3
29
​
30
[FILTER]
31
    Name nest
32
    Match *
33
    Operation lift
34
    Nested_under LAYER3
35
    Add_prefix Lifted3_
36
​
37
[FILTER]
38
    Name nest
39
    Match *
40
    Operation lift
41
    Nested_under Lifted3_LAYER2
42
    Add_prefix Lifted3_Lifted2_
43
​
44
[FILTER]
45
    Name nest
46
    Match *
47
    Operation lift
48
    Nested_under Lifted3_Lifted2_LAYER1
49
    Add_prefix Lifted3_Lifted2_Lifted1_
Copied!
Result
1
[0] mem.local: [1524862951.013414798, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996}]
2
​
3
​
4
{
5
  "Swap.total"=>1046524, 
6
  "Swap.used"=>0, 
7
  "Swap.free"=>1046524, 
8
  "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, 
9
  "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, 
10
  "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996
11
}
Copied!