Comment on page

Modify

The Modify Filter plugin allows you to change records using rules and conditions.
Example usage
As an example using JSON notation to,
Rename Key2 to RenamedKey
Add a key OtherKey with value Value3 if OtherKey does not yet exist
Example (input)
{
  "Key1"     : "Value1",
  "Key2"     : "Value2"
}
Example (output)
{
  "Key1"       : "Value1",
  "RenamedKey" : "Value2",
  "OtherKey"   : "Value3"
}
Configuration Parameters
Rules
The plugin supports the following rules:
Operation
Parameter 1
Parameter 2
Description
Set
STRING:KEY
STRING:VALUE
Add a key/value pair with key KEY and value VALUE. If KEY already exists, this field is overwritten
Add
STRING:KEY
STRING:VALUE
Add a key/value pair with key KEY and value VALUE if KEY does not exist
Remove
STRING:KEY
NONE
Remove a key/value pair with key KEY if it exists
Remove_wildcard
WILDCARD:KEY
NONE
Remove all key/value pairs with key matching wildcard KEY
Remove_regex
REGEXP:KEY
NONE
Remove all key/value pairs with key matching regexp KEY
Rename
STRING:KEY
STRING:RENAMED_KEY
Rename a key/value pair with key KEY to RENAMED_KEY if KEY exists AND RENAMED_KEY does not exist
Hard_rename
STRING:KEY
STRING:RENAMED_KEY
Rename a key/value pair with key KEY to RENAMED_KEY if KEY exists. If RENAMED_KEY already exists, this field is overwritten
Copy
STRING:KEY
STRING:COPIED_KEY
Copy a key/value pair with key KEY to COPIED_KEY if KEY exists AND COPIED_KEY does not exist
Hard_copy
STRING:KEY
STRING:COPIED_KEY
Copy a key/value pair with key KEY to COPIED_KEY if KEY exists. If COPIED_KEY already exists, this field is overwritten
Rules are case insensitive, parameters are not
Any number of rules can be set in a filter instance.
Rules are applied in the order they appear, with each rule operating on the result of the previous rule.
Conditions
The plugin supports the following conditions:
Condition
Parameter
Parameter 2
Description
Key_exists
STRING:KEY
NONE
Is true if KEY exists
Key_does_not_exist
STRING:KEY
STRING:VALUE
Is true if KEY does not exist
A_key_matches
REGEXP:KEY
NONE
Is true if a key matches regex KEY
No_key_matches
REGEXP:KEY
NONE
Is true if no key matches regex KEY
Key_value_equals
STRING:KEY
STRING:VALUE
Is true if KEY exists and its value is VALUE
Key_value_does_not_equal
STRING:KEY
STRING:VALUE
Is true if KEY exists and its value is not VALUE
Key_value_matches
STRING:KEY
REGEXP:VALUE
Is true if key KEY exists and its value matches VALUE
Key_value_does_not_match
STRING:KEY
REGEXP:VALUE
Is true if key KEY exists and its value does not match VALUE
Matching_keys_have_matching_values
REGEXP:KEY
REGEXP:VALUE
Is true if all keys matching KEY have values that match VALUE
Matching_keys_do_not_have_matching_values
REGEXP:KEY
REGEXP:VALUE
Is true if all keys matching KEY have values that do not match VALUE
Conditions are case insensitive, parameters are not
Any number of conditions can be set.
Conditions apply to the whole filter instance and all its rules. Not to individual rules.
All conditions have to be true for the rules to be applied.
Example #1 - Add and Rename
In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the Memory Usage Input Plugin, which outputs the following (example),
[0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
[1] memory: [1488543157, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
[2] memory: [1488543158, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
[3] memory: [1488543159, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
Using command Line
Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.
bin/fluent-bit -i mem \
  -p 'tag=mem.local' \
  -F modify \
  -p 'Add=Service1 SOMEVALUE' \
  -p 'Add=Service2 SOMEVALUE3' \
  -p 'Add=Mem.total2 TOTALMEM2' \
  -p 'Rename=Mem.free MEMFREE' \
  -p 'Rename=Mem.used MEMUSED' \
  -p 'Rename=Swap.total SWAPTOTAL' \
  -p 'Add=Mem.total TOTALMEM' \
  -m '*' \
  -o stdout
Configuration File
[INPUT]
    Name mem
    Tag  mem.local
​
[OUTPUT]
    Name  stdout
    Match *
​
[FILTER]
    Name modify
    Match *
    Add Service1 SOMEVALUE
    Add Service3 SOMEVALUE3
    Add Mem.total2 TOTALMEM2
    Rename Mem.free MEMFREE
    Rename Mem.used MEMUSED
    Rename Swap.total SWAPTOTAL
    Add Mem.total TOTALMEM
Result
The output of both the command line and configuration invocations should be identical and result in the following output.
[2018/04/06 01:35:13] [ info] [engine] started
[0] mem.local: [1522980610.006892802, {"Mem.total"=>4050908, "MEMUSED"=>738100, "MEMFREE"=>3312808, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
[1] mem.local: [1522980611.000658288, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
[2] mem.local: [1522980612.000307652, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
[3] mem.local: [1522980613.000122671, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
Example #2 - Conditionally Add and Remove
Configuration File
[INPUT]
    Name mem
    Tag  mem.local
    Interval_Sec 1
​
[FILTER]
    Name    modify
    Match   mem.*
​
    Condition Key_Does_Not_Exist cpustats
    Condition Key_Exists Mem.used
​
    Set cpustats UNKNOWN
​
[FILTER]
    Name    modify
    Match   mem.*
​
    Condition Key_Value_Does_Not_Equal cpustats KNOWN
​
    Add sourcetype memstats
​
[FILTER]
    Name    modify
    Match   mem.*
​
    Condition Key_Value_Equals cpustats UNKNOWN
​
    Remove_wildcard Mem
    Remove_wildcard Swap
    Add cpustats_more STILL_UNKNOWN
​
[OUTPUT]
    Name           stdout
    Match          *
Result
[2018/06/14 07:37:34] [ info] [engine] started (pid=1493)
[0] mem.local: [1528925855.000223110, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
[1] mem.local: [1528925856.000064516, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
[2] mem.local: [1528925857.000165965, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
[3] mem.local: [1528925858.000152319, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
Example #3 - Emoji
Configuration File
[INPUT]
    Name mem
    Tag  mem.local
​
[OUTPUT]
    Name  stdout
    Match *
​
[FILTER]
    Name modify
    Match *
​
    Remove_Wildcard Mem
    Remove_Wildcard Swap
    Set This_plugin_is_on 🔥
    Set 🔥 is_hot
    Copy 🔥 💦
    Rename  💦 ❄️
    Set ❄️ is_cold
    Set 💦 is_wet
Result
[2018/06/14 07:46:11] [ info] [engine] started (pid=21875)
[0] mem.local: [1528926372.000197916, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
[1] mem.local: [1528926373.000107868, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
[2] mem.local: [1528926374.000181042, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
[3] mem.local: [1528926375.000090841, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
[0] mem.local: [1528926376.000610974, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]

Last modified 4yr ago

Operation	Parameter 1	Parameter 2	Description
Set	STRING:KEY	STRING:VALUE	Add a key/value pair with key `KEY` and value `VALUE`. If `KEY` already exists, this field is overwritten
Add	STRING:KEY	STRING:VALUE	Add a key/value pair with key `KEY` and value `VALUE` if `KEY` does not exist
Remove	STRING:KEY	NONE	Remove a key/value pair with key `KEY` if it exists
Remove_wildcard	WILDCARD:KEY	NONE	Remove all key/value pairs with key matching wildcard `KEY`
Remove_regex	REGEXP:KEY	NONE	Remove all key/value pairs with key matching regexp `KEY`
Rename	STRING:KEY	STRING:RENAMED_KEY	Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists AND `RENAMED_KEY` does not exist
Hard_rename	STRING:KEY	STRING:RENAMED_KEY	Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists. If `RENAMED_KEY` already exists, this field is overwritten
Copy	STRING:KEY	STRING:COPIED_KEY	Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists AND `COPIED_KEY` does not exist
Hard_copy	STRING:KEY	STRING:COPIED_KEY	Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists. If `COPIED_KEY` already exists, this field is overwritten

Condition	Parameter	Parameter 2	Description
Key_exists	STRING:KEY	NONE	Is `true` if `KEY` exists
Key_does_not_exist	STRING:KEY	STRING:VALUE	Is `true` if `KEY` does not exist
A_key_matches	REGEXP:KEY	NONE	Is `true` if a key matches regex `KEY`
No_key_matches	REGEXP:KEY	NONE	Is `true` if no key matches regex `KEY`
Key_value_equals	STRING:KEY	STRING:VALUE	Is `true` if `KEY` exists and its value is `VALUE`
Key_value_does_not_equal	STRING:KEY	STRING:VALUE	Is `true` if `KEY` exists and its value is not `VALUE`
Key_value_matches	STRING:KEY	REGEXP:VALUE	Is `true` if key `KEY` exists and its value matches `VALUE`
Key_value_does_not_match	STRING:KEY	REGEXP:VALUE	Is `true` if key `KEY` exists and its value does not match `VALUE`
Matching_keys_have_matching_values	REGEXP:KEY	REGEXP:VALUE	Is `true` if all keys matching `KEY` have values that match `VALUE`
Matching_keys_do_not_have_matching_values	REGEXP:KEY	REGEXP:VALUE	Is `true` if all keys matching `KEY` have values that do not match `VALUE`