The Nest Filter plugin allows you to operate on or with nested data. Its modes of operation are
nest
- Take a set of records and place them in a map
lift
- Take a map by key and lift its records up
As an example using JSON notation, to nest keys matching the Wildcard
value Key*
under a new key NestKey
the transformation becomes,
Example (input)
{"Key1" : "Value1","Key2" : "Value2","OtherKey" : "Value3"}
Example (output)
{"OtherKey" : "Value3""NestKey" : {"Key1" : "Value1","Key2" : "Value2",}}
As an example using JSON notation, to lift keys nested under the Nested_under
value NestKey*
the transformation becomes,
Example (input)
{"OtherKey" : "Value3""NestKey" : {"Key1" : "Value1","Key2" : "Value2",}}
Example (output)
{"Key1" : "Value1","Key2" : "Value2","OtherKey" : "Value3"}
The plugin supports the following configuration parameters:
Key | Value Format | Operation | Description |
Operation | ENUM [ | ​ | Select the operation |
Wildcard | FIELD WILDCARD |
| Nest records which field matches the wildcard |
Nest_under | FIELD STRING |
| Nest records matching the |
Nested_under | FIELD STRING |
| Lift records nested under the |
Add_prefix | FIELD STRING | ANY | Prefix affected keys with this string |
Remove_prefix | FIELD STRING | ANY | Remove prefix from affected keys if it matches this string |
In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the Memory Usage Input Plugin, which outputs the following (example),
[0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.
The following command will load the mem plugin. Then the nest filter will match the wildcard rule to the keys and nest the keys matching Mem.*
under the new key NEST
.
$ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout
[INPUT]Name memTag mem.local​[OUTPUT]Name stdoutMatch *​[FILTER]Name nestMatch *Operation nestWildcard Mem.*Nest_under MemstatsRemove_prefix Mem.
The output of both the command line and configuration invocations should be identical and result in the following output.
[2018/04/06 01:35:13] [ info] [engine] started[0] mem.local: [1522978514.007359767, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Memstats"=>{"total"=>4050908, "used"=>714984, "free"=>3335924}}]
This example nests all Mem.*
and Swap,*
items under the Stats
key and then reverses these actions with a lift
operation. The output appears unchanged.
[INPUT]Name memTag mem.local​[OUTPUT]Name stdoutMatch *​[FILTER]Name nestMatch *Operation nestWildcard Mem.*Wildcard Swap.*Nest_under StatsAdd_prefix NESTED​[FILTER]Name nestMatch *Operation liftNested_under StatsRemove_prefix NESTED
[2018/06/21 17:42:37] [ info] [engine] started (pid=17285)[0] mem.local: [1529566958.000940636, {"Mem.total"=>8053656, "Mem.used"=>6940380, "Mem.free"=>1113276, "Swap.total"=>16532988, "Swap.used"=>1286772, "Swap.free"=>15246216}]
This example takes the keys starting with Mem.*
and nests them under LAYER1
, which itself is then nested under LAYER2
, which is nested under LAYER3
.
[INPUT]Name memTag mem.local​[OUTPUT]Name stdoutMatch *​[FILTER]Name nestMatch *Operation nestWildcard Mem.*Nest_under LAYER1​[FILTER]Name nestMatch *Operation nestWildcard LAYER1*Nest_under LAYER2​[FILTER]Name nestMatch *Operation nestWildcard LAYER2*Nest_under LAYER3
[0] mem.local: [1524795923.009867831, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "LAYER3"=>{"LAYER2"=>{"LAYER1"=>{"Mem.total"=>4050908, "Mem.used"=>1112036, "Mem.free"=>2938872}}}}]​​{"Swap.total"=>1046524,"Swap.used"=>0,"Swap.free"=>1046524,"LAYER3"=>{"LAYER2"=>{"LAYER1"=>{"Mem.total"=>4050908,"Mem.used"=>1112036,"Mem.free"=>2938872}}}}
This example starts with the 3-level deep nesting of Example 2 and applies the lift
filter three times to reverse the operations. The end result is that all records are at the top level, without nesting, again. One prefix is added for each level that is lifted.
[INPUT]Name memTag mem.local​[OUTPUT]Name stdoutMatch *​[FILTER]Name nestMatch *Operation nestWildcard Mem.*Nest_under LAYER1​[FILTER]Name nestMatch *Operation nestWildcard LAYER1*Nest_under LAYER2​[FILTER]Name nestMatch *Operation nestWildcard LAYER2*Nest_under LAYER3​[FILTER]Name nestMatch *Operation liftNested_under LAYER3Add_prefix Lifted3_​[FILTER]Name nestMatch *Operation liftNested_under Lifted3_LAYER2Add_prefix Lifted3_Lifted2_​[FILTER]Name nestMatch *Operation liftNested_under Lifted3_Lifted2_LAYER1Add_prefix Lifted3_Lifted2_Lifted1_
[0] mem.local: [1524862951.013414798, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996}]​​{"Swap.total"=>1046524,"Swap.used"=>0,"Swap.free"=>1046524,"Lifted3_Lifted2_Lifted1_Mem.total"=>4050908,"Lifted3_Lifted2_Lifted1_Mem.used"=>1253912,"Lifted3_Lifted2_Lifted1_Mem.free"=>2796996}