# Modify The *Modify Filter* plugin lets you change records using rules and conditions. ## Example usage As an example, use JSON notation to: * Rename `Key2` to `RenamedKey` * Add a key `OtherKey` with value `Value3` if `OtherKey` doesn't yet exist Example (input): ``` { "Key1" : "Value1", "Key2" : "Value2" } ``` Example (output): ``` { "Key1" : "Value1", "RenamedKey" : "Value2", "OtherKey" : "Value3" } ``` ## Configuration parameters The Modify filter supports multiple parameters. ### Rules The plugin supports the following rules: | Operation | Parameter 1 | Parameter 2 | Description | | ----------------- | -------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `Set` | `STRING:KEY` | `STRING:VALUE` | Add a key/value pair with key `KEY` and value `VALUE`. If `KEY` already exists, this field is overwritten. | | `Add` | `STRING:KEY` | `STRING:VALUE` | Add a key/value pair with key `KEY` and value `VALUE` if `KEY` doesn't exist. | | `Remove` | `STRING:KEY` | *none* | Remove a key/value pair with key `KEY` if it exists. | | `Remove_wildcard` | `WILDCARD:KEY` | *none* | Remove all key/value pairs with key matching wildcard `KEY`. | | `Remove_regex` | `REGEXP:KEY` | *none* | Remove all key/value pairs with key matching regexp `KEY`. | | `Rename` | `STRING:KEY` | `STRING:RENAMED_KEY` | Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists and `RENAMED_KEY` doesn't exist. | | `Hard_rename` | `STRING:KEY` | `STRING:RENAMED_KEY` | Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists. If `RENAMED_KEY` already exists, this field is overwritten. | | `Copy` | `STRING:KEY` | `STRING:COPIED_KEY` | Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists and `COPIED_KEY` doesn't exist. | | `Hard_copy` | `STRING:KEY` | `STRING:COPIED_KEY` | Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists. If `COPIED_KEY` already exists, this field is overwritten. | | `Move_to_start` | `WILDCARD:KEY` | *none* | Move key/value pairs with keys matching `KEY` to the start of the message. | | `Move_to_end` | `WILDCARD:KEY` | *none* | Move key/value pairs with keys matching `KEY` to the end of the message. | * Rules are case-insensitive, but parameters aren't. * Any number of rules can be set in a filter instance. * Rules are applied in the order they appear, with each rule operating on the result of the previous rule. ### Conditions The plugin supports the following conditions: | Condition | Parameter | Parameter 2 | Description | | ------------------------------------------- | ------------ | -------------- | -------------------------------------------------------------------------- | | `Key_exists` | `STRING:KEY` | *none* | Is `true` if `KEY` exists. | | `Key_does_not_exist` | `STRING:KEY` | *none* | Is `true` if `KEY` doesn't exist. | | `A_key_matches` | `REGEXP:KEY` | *none* | Is `true` if a key matches regular expression `KEY`. | | `No_key_matches` | `REGEXP:KEY` | *none* | Is `true` if no key matches regular expression `KEY`. | | `Key_value_equals` | `STRING:KEY` | `STRING:VALUE` | Is `true` if `KEY` exists and its value is `VALUE`. | | `Key_value_does_not_equal` | `STRING:KEY` | `STRING:VALUE` | Is `true` if `KEY` exists and its value isn't `VALUE`. | | `Key_value_matches` | `STRING:KEY` | `REGEXP:VALUE` | Is `true` if key `KEY` exists and its value matches `VALUE`. | | `Key_value_does_not_match` | `STRING:KEY` | `REGEXP:VALUE` | Is `true` if key `KEY` exists and its value doesn't match `VALUE`. | | `Matching_keys_have_matching_values` | `REGEXP:KEY` | `REGEXP:VALUE` | Is `true` if all keys matching `KEY` have values that match `VALUE`. | | `Matching_keys_do_not_have_matching_values` | `REGEXP:KEY` | `REGEXP:VALUE` | Is `true` if all keys matching `KEY` have values that don't match `VALUE`. | * Conditions are case-insensitive, but parameters aren't. * Any number of conditions can be set. * Conditions apply to the whole filter instance and all its rules. *Not* to individual rules. * All conditions must be `true` for the rules to be applied. * You can set [Record Accessor](https://docs.fluentbit.io/manual/4.0/administration/configuring-fluent-bit/classic-mode/record-accessor) as `STRING:KEY` for nested key. ## Example 1 - add and rename To start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the [Memory Usage Input Plugin](https://docs.fluentbit.io/manual/4.0/data-pipeline/inputs/memory-metrics), which outputs data similar to the following: ``` [0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}] [1] memory: [1488543157, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}] [2] memory: [1488543158, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}] [3] memory: [1488543159, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}] ``` ### Using command line Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended. ```shell fluent-bit -i mem \ -p 'tag=mem.local' \ -F modify \ -p 'Add=Service1 SOMEVALUE' \ -p 'Add=Service2 SOMEVALUE3' \ -p 'Add=Mem.total2 TOTALMEM2' \ -p 'Rename=Mem.free MEMFREE' \ -p 'Rename=Mem.used MEMUSED' \ -p 'Rename=Swap.total SWAPTOTAL' \ -p 'Add=Mem.total TOTALMEM' \ -m '*' \ -o stdout ``` ### Configuration file {% tabs %} {% tab title="fluent-bit.yaml" %} ```yaml pipeline: inputs: - name: mem tag: mem.local filters: - name: modify match: '*' Add: - Service1 SOMEVALUE - Service3 SOMEVALUE3 - Mem.total2 TOTALMEM2 - Mem.total TOTALMEM Rename: - Mem.free MEMFREE - Mem.used MEMUSED - Swap.total SWAPTOTAL outputs: - name: stdout match: '*' ``` {% endtab %} {% tab title="fluent-bit.conf" %} ``` [INPUT] Name mem Tag mem.local [FILTER] Name modify Match * Add Service1 SOMEVALUE Add Service3 SOMEVALUE3 Add Mem.total2 TOTALMEM2 Rename Mem.free MEMFREE Rename Mem.used MEMUSED Rename Swap.total SWAPTOTAL Add Mem.total TOTALMEM [OUTPUT] Name stdout Match * ``` {% endtab %} {% endtabs %} ### Result The output of both the command line and configuration invocations should be identical and result in the following output: ``` [2018/04/06 01:35:13] [ info] [engine] started [0] mem.local: [1522980610.006892802, {"Mem.total"=>4050908, "MEMUSED"=>738100, "MEMFREE"=>3312808, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}] [1] mem.local: [1522980611.000658288, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}] [2] mem.local: [1522980612.000307652, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}] [3] mem.local: [1522980613.000122671, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}] ``` ## Example 2 - conditionally add and remove ### Use a configuration file {% tabs %} {% tab title="fluent-bit.yaml" %} ```yaml pipeline: inputs: - name: mem tag: mem.local interval_sec: 1 filters: - name: modify match: mem.* Condition: - Key_Does_Not_Exist cpustats - Key_Exists Mem.used Set: cpustats UNKNOWN - name: modify match: mem.* Condition: Key_Value_Does_Not_Equal cpustats KNOWN Add: sourcetype memstats - name: modify match: mem.* Condition: Key_Value_Equals cpustats UNKNOWN Remove_wildcard: - Mem - Swap Add: cpustats_more STILL_UNKNOWN outputs: - name: stdout match: '*' ``` {% endtab %} {% tab title="fluent-bit.conf" %} ``` [INPUT] Name mem Tag mem.local Interval_Sec 1 [FILTER] Name modify Match mem.* Condition Key_Does_Not_Exist cpustats Condition Key_Exists Mem.used Set cpustats UNKNOWN [FILTER] Name modify Match mem.* Condition Key_Value_Does_Not_Equal cpustats KNOWN Add sourcetype memstats [FILTER] Name modify Match mem.* Condition Key_Value_Equals cpustats UNKNOWN Remove_wildcard Mem Remove_wildcard Swap Add cpustats_more STILL_UNKNOWN [OUTPUT] Name stdout Match * ``` {% endtab %} {% endtabs %} ### Add and remove result ``` [2018/06/14 07:37:34] [ info] [engine] started (pid=1493) [0] mem.local: [1528925855.000223110, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}] [1] mem.local: [1528925856.000064516, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}] [2] mem.local: [1528925857.000165965, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}] [3] mem.local: [1528925858.000152319, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}] ``` ## Example 3 - emoji ### Emoji configuration file {% tabs %} {% tab title="fluent-bit.yaml" %} ```yaml pipeline: inputs: - name: mem tag: mem.local interval_sec: 1 filters: - name: modify match: mem.* Remove_wildcard: - Mem - Swap set: - This_plugin_is_on 🔥 - 🔥 is_hot - ❄️ is_cold - 💦 is_wet copy: 🔥 💦 rename: 💦 ❄️ outputs: - name: stdout match: '*' ``` {% endtab %} {% tab title="fluent-bit.conf" %} ``` [INPUT] Name mem Tag mem.local [FILTER] Name modify Match * Remove_Wildcard Mem Remove_Wildcard Swap Set This_plugin_is_on 🔥 Set 🔥 is_hot Copy 🔥 💦 Rename 💦 ❄️ Set ❄️ is_cold Set 💦 is_wet [OUTPUT] Name stdout Match * ``` {% endtab %} {% endtabs %} ### Emoji example result ``` [2018/06/14 07:46:11] [ info] [engine] started (pid=21875) [0] mem.local: [1528926372.000197916, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}] [1] mem.local: [1528926373.000107868, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}] [2] mem.local: [1528926374.000181042, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}] [3] mem.local: [1528926375.000090841, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}] [0] mem.local: [1528926376.000610974, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}] ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.fluentbit.io/manual/4.0/data-pipeline/filters/modify.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.