The Modify Filter plugin allows you to change records using rules and conditions.
Example usage
As an example using JSON notation to,
Rename Key2
to RenamedKey
Add a key OtherKey
with value Value3
if OtherKey
does not yet exist
Example (input)
Copy {
"Key1" : "Value1",
"Key2" : "Value2"
}
Example (output)
Copy {
"Key1" : "Value1",
"RenamedKey" : "Value2",
"OtherKey" : "Value3"
}
Configuration Parameters
Rules
The plugin supports the following rules:
Operation Parameter 1 Parameter 2 Description Add a key/value pair with key KEY
and value VALUE
. If KEY
already exists, this field is overwritten
Add a key/value pair with key KEY
and value VALUE
if KEY
does not exist
Remove a key/value pair with key KEY
if it exists
Remove all key/value pairs with key matching wildcard KEY
Remove all key/value pairs with key matching regexp KEY
Rename a key/value pair with key KEY
to RENAMED_KEY
if KEY
exists AND RENAMED_KEY
does not exist
Rename a key/value pair with key KEY
to RENAMED_KEY
if KEY
exists. If RENAMED_KEY
already exists, this field is overwritten
Copy a key/value pair with key KEY
to COPIED_KEY
if KEY
exists AND COPIED_KEY
does not exist
Copy a key/value pair with key KEY
to COPIED_KEY
if KEY
exists. If COPIED_KEY
already exists, this field is overwritten
Move key/value pairs with keys matching KEY to the start of the message
Move key/value pairs with keys matching KEY to the end of the message
Rules are case insensitive, parameters are not
Any number of rules can be set in a filter instance.
Rules are applied in the order they appear, with each rule operating on the result of the previous rule.
Conditions
The plugin supports the following conditions:
Condition Parameter Parameter 2 Description Is true
if KEY
does not exist
Is true
if a key matches regex KEY
Is true
if no key matches regex KEY
Is true
if KEY
exists and its value is VALUE
Is true
if KEY
exists and its value is not VALUE
Is true
if key KEY
exists and its value matches VALUE
Is true
if key KEY
exists and its value does not match VALUE
Matching_keys_have_matching_values
Is true
if all keys matching KEY
have values that match VALUE
Matching_keys_do_not_have_matching_values
Is true
if all keys matching KEY
have values that do not match VALUE
Conditions are case insensitive, parameters are not
Any number of conditions can be set.
Conditions apply to the whole filter instance and all its rules. Not to individual rules.
All conditions have to be true
for the rules to be applied.
Example #1 - Add and Rename
In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the Memory Usage Input Plugin , which outputs the following (example),
Copy [0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
[1] memory: [1488543157, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
[2] memory: [1488543158, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
[3] memory: [1488543159, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
Using command Line
Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.
Copy bin/fluent-bit -i mem \
-p 'tag=mem.local' \
-F modify \
-p 'Add=Service1 SOMEVALUE' \
-p 'Add=Service2 SOMEVALUE3' \
-p 'Add=Mem.total2 TOTALMEM2' \
-p 'Rename=Mem.free MEMFREE' \
-p 'Rename=Mem.used MEMUSED' \
-p 'Rename=Swap.total SWAPTOTAL' \
-p 'Add=Mem.total TOTALMEM' \
-m '*' \
-o stdout
Configuration File
fluent-bit.conf fluent-bit.yaml
Copy [INPUT]
Name mem
Tag mem . local
[OUTPUT]
Name stdout
Match *
[FILTER]
Name modify
Match *
Add Service1 SOMEVALUE
Add Service3 SOMEVALUE3
Add Mem . total2 TOTALMEM2
Rename Mem . free MEMFREE
Rename Mem . used MEMUSED
Rename Swap . total SWAPTOTAL
Add Mem . total TOTALMEM
Copy pipeline :
inputs :
- name : mem
tag : mem.local
filters :
- name : modify
match : '*'
Add :
- Service1 SOMEVALUE
- Service3 SOMEVALUE3
- Mem.total2 TOTALMEM2
- Mem.total TOTALMEM
Rename :
- Mem.free MEMFREE
- Mem.used MEMUSED
- Swap.total SWAPTOTAL
outputs :
- name : stdout
match : '*'
Result
The output of both the command line and configuration invocations should be identical and result in the following output.
Copy [2018/04/06 01:35:13] [ info] [engine] started
[0] mem.local: [1522980610.006892802, {"Mem.total"=>4050908, "MEMUSED"=>738100, "MEMFREE"=>3312808, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
[1] mem.local: [1522980611.000658288, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
[2] mem.local: [1522980612.000307652, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
[3] mem.local: [1522980613.000122671, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
Example #2 - Conditionally Add and Remove
Configuration File
fluent-bit.conf fluent-bit.yaml
Copy [INPUT]
Name mem
Tag mem . local
Interval_Sec 1
[FILTER]
Name modify
Match mem . *
Condition Key_Does_Not_Exist cpustats
Condition Key_Exists Mem . used
Set cpustats UNKNOWN
[FILTER]
Name modify
Match mem . *
Condition Key_Value_Does_Not_Equal cpustats KNOWN
Add sourcetype memstats
[FILTER]
Name modify
Match mem . *
Condition Key_Value_Equals cpustats UNKNOWN
Remove_wildcard Mem
Remove_wildcard Swap
Add cpustats_more STILL_UNKNOWN
[OUTPUT]
Name stdout
Match *
Copy pipeline :
inputs :
- name : mem
tag : mem.local
interval_sec : 1
filters :
- name : modify
match : mem.*
Condition :
- Key_Does_Not_Exist cpustats
- Key_Exists Mem.used
Set : cpustats UNKNOWN
- name : modify
match : mem.*
Condition : Key_Value_Does_Not_Equal cpustats KNOWN
Add : sourcetype memstats
- name : modify
match : mem.*
Condition : Key_Value_Equals cpustats UNKNOWN
Remove_wildcard :
- Mem
- Swap
Add : cpustats_more STILL_UNKNOWN
outputs :
- name : stdout
match : '*'
Result
Copy [2018/06/14 07:37:34] [ info] [engine] started (pid=1493)
[0] mem.local: [1528925855.000223110, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
[1] mem.local: [1528925856.000064516, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
[2] mem.local: [1528925857.000165965, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
[3] mem.local: [1528925858.000152319, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
Example #3 - Emoji
Configuration File
fluent-bit.conf fluent-bit.yaml
Copy [INPUT]
Name mem
Tag mem . local
[OUTPUT]
Name stdout
Match *
[FILTER]
Name modify
Match *
Remove_Wildcard Mem
Remove_Wildcard Swap
Set This_plugin_is_on 🔥
Set 🔥 is_hot
Copy 🔥 💦
Rename 💦 ❄️
Set ❄️ is_cold
Set 💦 is_wet
Copy pipeline :
inputs :
- name : mem
tag : mem.local
interval_sec : 1
filters :
- name : modify
match : mem.*
Remove_wildcard :
- Mem
- Swap
Set :
- This_plugin_is_on 🔥
- 🔥 is_hot
Copy : 🔥 💦
Rename : 💦 ❄️
Set :
- ❄️ is_cold
- 💦 is_wet
outputs :
- name : stdout
match : '*'
Result
Copy [2018/06/14 07:46:11] [ info] [engine] started (pid=21875)
[0] mem.local: [1528926372.000197916, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
[1] mem.local: [1528926373.000107868, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
[2] mem.local: [1528926374.000181042, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
[3] mem.local: [1528926375.000090841, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
[0] mem.local: [1528926376.000610974, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
Last updated 2 months ago