# Parser The *Parser* filter allows for parsing fields in event records. ## Configuration parameters The plugin supports the following configuration parameters: | Key | Description | Default | | -------------- | --------------------------------------------------------------------------------------------------------- | ------- | | `Key_Name` | Specify field name in record to parse. | *none* | | `Parser` | Specify the parser name to interpret the field. Multiple parser entries are allowed (one per line). | *none* | | `Preserve_Key` | Keep the original `Key_Name` field in the parsed result. If false, the field will be removed. | `False` | | `Reserve_Data` | Keep all other original fields in the parsed result. If false, all other original fields will be removed. | `False` | ## Get started ### Configuration file The plugin needs a parser file which defines how to parse each field. This is an example of parsing a record `{"data":"100 0.5 true This is example"}`. {% tabs %} {% tab title="fluent-bit.yaml" %} ```yaml parsers: - name: dummy_test format: regex regex: '^(?[^ ]+) (?[^ ]+) (?[^ ]+) (?.+)$' ``` {% endtab %} {% tab title="fluent-bit.conf" %} ``` [PARSER] Name dummy_test Format regex Regex ^(?[^ ]+) (?[^ ]+) (?[^ ]+) (?.+)$ ``` {% endtab %} {% endtabs %} The path of the parser file should be written in configuration file under the `[SERVICE]` section. {% tabs %} {% tab title="fluent-bit.yaml" %} ```yaml service: parsers_file: /path/to/parsers.yaml pipeline: inputs: - name: dummy tag: dummy.data dummy: '{"data":"100 0.5 true This is example"}' filters: - name: parser match: 'dummy.*' key_name: data parser: dummy_test outputs: - name: stdout match: '*' ``` {% endtab %} {% tab title="fluent-bit.conf" %} ``` [SERVICE] Parsers_File /path/to/parsers.conf [INPUT] Name dummy Tag dummy.data Dummy {"data":"100 0.5 true This is example"} [FILTER] Name parser Match dummy.* Key_Name data Parser dummy_test [OUTPUT] Name stdout Match * ``` {% endtab %} {% endtabs %} The output when running the corresponding configuration is as follows: ``` # For YAML configuration. $ fluent-bit --config fluent-bit.yaml # For classic configuration. $ fluent-bit --config fluent-bit.conf ... [0] dummy.data: [[1750323528.603308000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}] [0] dummy.data: [[1750323529.603788000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}] [0] dummy.data: [[1750323530.604204000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}] [0] dummy.data: [[1750323531.603961000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]e ``` You can see the records `{"data":"100 0.5 true This is example"}` are parsed. ### Preserve original fields By default, the parser plugin only keeps the parsed fields in its output. If you enable `Reserve_Data`, all other fields are preserved. First the contents of the corresponding parsers file, depending on the choice for YAML or classic configurations, would be as follows: {% tabs %} {% tab title="parsers.yaml" %} ```yaml parsers: - name: dummy_test format: regex regex: '^(?[^ ]+) (?[^ ]+) (?[^ ]+) (?.+)$' ``` {% endtab %} {% tab title="parsers.conf" %} ``` [PARSER] Name dummy_test Format regex Regex ^(?[^ ]+) (?[^ ]+) (?[^ ]+) (?.+)$ ``` {% endtab %} {% endtabs %} Now add `Reserve_Data` to the filter section of the corresponding configuration file as follows: {% tabs %} {% tab title="fluent-bit.yaml" %} ```yaml service: parsers_file: /path/to/parsers.yaml pipeline: inputs: - name: dummy tag: dummy.data dummy: '{"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"}' filters: - name: parser match: 'dummy.*' key_name: data parser: dummy_test reserve_data: on outputs: - name: stdout match: '*' ``` {% endtab %} {% tab title="fluent-bit.conf" %} ``` [SERVICE] Parsers_File /path/to/parsers.conf [INPUT] Name dummy Tag dummy.data Dummy {"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"} [FILTER] Name parser Match dummy.* Key_Name data Parser dummy_test Reserve_Data On [OUTPUT] Name stdout Match * ``` {% endtab %} {% endtabs %} The output when running the corresponding configuration is as follows: ``` # For YAML configuration. $ fluent-bit --config fluent-bit.yaml # For classic configuration. $ fluent-bit --config fluent-bit.conf ... [0] dummy.data: [[1750325238.681398000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "key1"=>"value1", "key2"=>"value2"}] [0] dummy.data: [[1750325239.682090000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "key1"=>"value1", "key2"=>"value2"}] [0] dummy.data: [[1750325240.682903000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "key1"=>"value1", "key2"=>"value2"}] ``` If you enable `Reserve_Data` and `Preserve_Key`, the original key field will also be preserved. First the contents of the corresponding parsers file, depending on the choice for YAML or classic configurations, would be as follows: {% tabs %} {% tab title="parsers.yaml" %} ```yaml parsers: - name: dummy_test format: regex regex: '^(?[^ ]+) (?[^ ]+) (?[^ ]+) (?.+)$' ``` {% endtab %} {% tab title="parsers.conf" %} ``` [PARSER] Name dummy_test Format regex Regex ^(?[^ ]+) (?[^ ]+) (?[^ ]+) (?.+)$ ``` {% endtab %} {% endtabs %} Now add `Reserve_Data` and `Preserve_Key`to the filter section of the corresponding configuration file as follows: {% tabs %} {% tab title="fluent-bit.yaml" %} ```yaml service: parsers_file: /path/to/parsers.yaml pipeline: inputs: - name: dummy tag: dummy.data dummy: '{"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"}' filters: - name: parser match: 'dummy.*' key_name: data parser: dummy_test reserve_data: on preserve_key: on outputs: - name: stdout match: '*' ``` {% endtab %} {% tab title="fluent-bit.conf" %} ``` [SERVICE] Parsers_File /path/to/parsers.conf [INPUT] Name dummy Tag dummy.data Dummy {"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"} [FILTER] Name parser Match dummy.* Key_Name data Parser dummy_test Reserve_Data On Preserve_Key On [OUTPUT] Name stdout Match * ``` {% endtab %} {% endtabs %} The output when running the corresponding configuration is as follows: ``` # For YAML configuration. $ fluent-bit --config fluent-bit.yaml # For classic configuration. $ fluent-bit --config fluent-bit.conf ... [0] dummy.data: [[1750325678.572817000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "data"=>"100 0.5 true This is example", "key1"=>"value1", "key2"=>"value2"}] [0] dummy.data: [[1750325679.574538000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "data"=>"100 0.5 true This is example", "key1"=>"value1", "key2"=>"value2"}] [0] dummy.data: [[1750325680.569750000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "data"=>"100 0.5 true This is example", "key1"=>"value1", "key2"=>"value2"}] ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.fluentbit.io/manual/4.0/data-pipeline/filters/parser.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.