Modify

Supported event types: logs

The Modify Filter plugin lets you change records using rules and conditions.

Example usage

As an example, use JSON notation to:

• Rename Key2 to RenamedKey

• Add a key OtherKey with value Value3 if OtherKey doesn't yet exist

Example (input):

{
  "Key1"     : "Value1",
  "Key2"     : "Value2"
}

Example (output):

{
  "Key1"       : "Value1",
  "RenamedKey" : "Value2",
  "OtherKey"   : "Value3"
}

Configuration parameters

The Modify filter supports multiple parameters.

Rules

The plugin supports the following rules:

Operation	Parameter 1	Parameter 2	Description
Set	STRING:KEY	STRING:VALUE	Add a key/value pair with key KEY and value VALUE. If KEY already exists, this field is overwritten.
Add	STRING:KEY	STRING:VALUE	Add a key/value pair with key KEY and value VALUE if KEY doesn't exist.
Remove	STRING:KEY	none	Remove a key/value pair with key KEY if it exists.
Remove_wildcard	WILDCARD:KEY	none	Remove all key/value pairs with key matching wildcard KEY.
Remove_regex	REGEXP:KEY	none	Remove all key/value pairs with key matching regexp KEY.
Rename	STRING:KEY	STRING:RENAMED_KEY	Rename a key/value pair with key KEY to RENAMED_KEY if KEY exists and RENAMED_KEY doesn't exist.
Hard_rename	STRING:KEY	STRING:RENAMED_KEY	Rename a key/value pair with key KEY to RENAMED_KEY if KEY exists. If RENAMED_KEY already exists, this field is overwritten.
Hard_Rename	STRING:KEY	STRING:RENAMED_KEY	Equivalent to Hard_rename. This spelling is also accepted by the plugin.
Copy	STRING:KEY	STRING:COPIED_KEY	Copy a key/value pair with key KEY to COPIED_KEY if KEY exists and COPIED_KEY doesn't exist.
Hard_copy	STRING:KEY	STRING:COPIED_KEY	Copy a key/value pair with key KEY to COPIED_KEY if KEY exists. If COPIED_KEY already exists, this field is overwritten.
Move_to_start	WILDCARD:KEY	none	Move key/value pairs with keys matching KEY to the start of the message.
Move_To_Start	WILDCARD:KEY	none	Equivalent to Move_to_start. This spelling is also accepted by the plugin.
Move_to_end	WILDCARD:KEY	none	Move key/value pairs with keys matching KEY to the end of the message.
Move_To_End	WILDCARD:KEY	none	Equivalent to Move_to_end. This spelling is also accepted by the plugin.

• Rules are case-insensitive, but parameters aren't.

• Any number of rules can be set in a filter instance.

• Rules are applied in the order they appear, with each rule operating on the result of the previous rule.

Conditions

The plugin supports the following conditions:

Condition	Parameter	Parameter 2	Description
Key_exists	STRING:KEY	none	Is true if KEY exists.
Key_does_not_exist	STRING:KEY	none	Is true if KEY doesn't exist.
A_key_matches	REGEXP:KEY	none	Is true if a key matches regular expression KEY.
No_key_matches	REGEXP:KEY	none	Is true if no key matches regular expression KEY.
Key_value_equals	STRING:KEY	STRING:VALUE	Is true if KEY exists and its value is VALUE.
Key_value_does_not_equal	STRING:KEY	STRING:VALUE	Is true if KEY exists and its value isn't VALUE.
Key_value_matches	STRING:KEY	REGEXP:VALUE	Is true if key KEY exists and its value matches VALUE.
Key_value_does_not_match	STRING:KEY	REGEXP:VALUE	Is true if key KEY exists and its value doesn't match VALUE.
Matching_keys_have_matching_values	REGEXP:KEY	REGEXP:VALUE	Is true if all keys matching KEY have values that match VALUE.
Matching_keys_do_not_have_matching_values	REGEXP:KEY	REGEXP:VALUE	Is true if all keys matching KEY have values that don't match VALUE.

• Conditions are case-insensitive, but parameters aren't.

• Any number of conditions can be set.

• Conditions apply to the whole filter instance and all its rules. Not to individual rules.

• All conditions must be true for the rules to be applied.

• You can set Record Accessor as STRING:KEY for nested key.

Example 1 - add and rename

To start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the Memory Usage Input Plugin, which outputs data similar to the following:

[0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
[1] memory: [1488543157, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
[2] memory: [1488543158, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
[3] memory: [1488543159, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]

Using command line

Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.

fluent-bit -i mem \
           -p 'tag=mem.local' \
           -F modify \
           -p 'Add=Service1 SOMEVALUE' \
           -p 'Add=Service2 SOMEVALUE3' \
           -p 'Add=Mem.total2 TOTALMEM2' \
           -p 'Rename=Mem.free MEMFREE' \
           -p 'Rename=Mem.used MEMUSED' \
           -p 'Rename=Swap.total SWAPTOTAL' \
           -p 'Add=Mem.total TOTALMEM' \
           -m '*' \
           -o stdout

Configuration file

fluent-bit.yaml

pipeline:
  inputs:
    - name: mem
      tag: mem.local

  filters:
    - name: modify
      match: '*'
      Add:
        - Service1 SOMEVALUE
        - Service3 SOMEVALUE3
        - Mem.total2 TOTALMEM2
        - Mem.total TOTALMEM
      Rename:
        - Mem.free MEMFREE
        - Mem.used MEMUSED
        - Swap.total SWAPTOTAL

  outputs:
    - name: stdout
      match: '*'

fluent-bit.conf

Result

The output of both the command line and configuration invocations should be identical and result in the following output:

[2018/04/06 01:35:13] [ info] [engine] started
[0] mem.local: [1522980610.006892802, {"Mem.total"=>4050908, "MEMUSED"=>738100, "MEMFREE"=>3312808, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
[1] mem.local: [1522980611.000658288, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
[2] mem.local: [1522980612.000307652, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
[3] mem.local: [1522980613.000122671, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]

Example 2 - conditionally add and remove

Use a configuration file

fluent-bit.yaml

pipeline:
  inputs:
    - name: mem
      tag: mem.local
      interval_sec: 1

  filters:
    - name: modify
      match: mem.*
      Condition:
        - Key_Does_Not_Exist cpustats
        - Key_Exists Mem.used
      Set: cpustats UNKNOWN

    - name: modify
      match: mem.*
      Condition: Key_Value_Does_Not_Equal cpustats KNOWN
      Add: sourcetype memstats

    - name: modify
      match: mem.*
      Condition: Key_Value_Equals cpustats UNKNOWN
      Remove_wildcard:
        - Mem
        - Swap
      Add: cpustats_more STILL_UNKNOWN

  outputs:
    - name: stdout
      match: '*'

fluent-bit.conf

Add and remove result

[2018/06/14 07:37:34] [ info] [engine] started (pid=1493)
[0] mem.local: [1528925855.000223110, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
[1] mem.local: [1528925856.000064516, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
[2] mem.local: [1528925857.000165965, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
[3] mem.local: [1528925858.000152319, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]

Example 3 - emoji

Emoji configuration file

fluent-bit.yaml

pipeline:
  inputs:
    - name: mem
      tag: mem.local
      interval_sec: 1

  filters:
    - name: modify
      match: mem.*
      Remove_wildcard:
        - Mem
        - Swap
      set:
        - This_plugin_is_on 🔥
        - 🔥 is_hot
        - ❄️ is_cold
        - 💦 is_wet
      copy: 🔥 💦
      rename:  💦 ❄️

  outputs:
    - name: stdout
      match: '*'

fluent-bit.conf

Emoji example result

[2018/06/14 07:46:11] [ info] [engine] started (pid=21875)
[0] mem.local: [1528926372.000197916, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
[1] mem.local: [1528926373.000107868, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
[2] mem.local: [1528926374.000181042, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
[3] mem.local: [1528926375.000090841, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
[0] mem.local: [1528926376.000610974, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]