The Parser Filter plugin allows for parsing fields in event records.
The plugin supports the following configuration parameters:
Key | Description | Default |
Key_Name | Specify field name in record to parse. | ​ |
Parser | Specify the parser name to interpret the field. Multiple Parser entries are allowed (one per line). | ​ |
Preserve_Key | Keep original | False |
Reserve_Data | Keep all other original fields in the parsed result. If false, all other original fields will be removed. | False |
Unescape_Key | If the key is a escaped string (e.g: stringify JSON), unescape the string before to apply the parser. | False |
This is an example of parsing a record {"data":"100 0.5 true This is example"}
.
The plugin needs a parser file which defines how to parse each field.
[PARSER]Name dummy_testFormat regexRegex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$
The path of the parser file should be written in configuration file under the [SERVICE] section.
[SERVICE]Parsers_File /path/to/parsers.conf​[INPUT]Name dummyTag dummy.dataDummy {"data":"100 0.5 true This is example"}​[FILTER]Name parserMatch dummy.*Key_Name dataParser dummy_test​[OUTPUT]Name stdoutMatch *
The output is
$ fluent-bit -c dummy.confFluent Bit v1.x.x* Copyright (C) 2019-2020 The Fluent Bit Authors* Copyright (C) 2015-2018 Treasure Data* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd* https://fluentbit.io​[2017/07/06 22:33:12] [ info] [engine] started[0] dummy.data: [1499347993.001371317, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}][1] dummy.data: [1499347994.001303118, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}][2] dummy.data: [1499347995.001296133, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}][3] dummy.data: [1499347996.001320284, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
You can see the records {"data":"100 0.5 true This is example"}
are parsed.
By default, the parser plugin only keeps the parsed fields in its output.
If you enable Reserve_Data
, all other fields are preserved:
[PARSER]Name dummy_testFormat regexRegex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$
[SERVICE]Parsers_File /path/to/parsers.conf​[INPUT]Name dummyTag dummy.dataDummy {"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"}​[FILTER]Name parserMatch dummy.*Key_Name dataParser dummy_testReserve_Data On
This will produce the output:
$ fluent-bit -c dummy.confFluent-Bit v0.12.0Copyright (C) Treasure Data​[2017/07/06 22:33:12] [ info] [engine] started[0] dummy.data: [1499347993.001371317, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"][1] dummy.data: [1499347994.001303118, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"][2] dummy.data: [1499347995.001296133, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"][3] dummy.data: [1499347996.001320284, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
If you enable Reserved_Data
and Preserve_Key
, the original key field will be preserved as well:
[PARSER]Name dummy_testFormat regexRegex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$
[SERVICE]Parsers_File /path/to/parsers.conf​[INPUT]Name dummyTag dummy.dataDummy {"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"}​[FILTER]Name parserMatch dummy.*Key_Name dataParser dummy_testReserve_Data OnPreserve_Key On
This will produce the following output:
$ fluent-bit -c dummy.confFluent-Bit v0.12.0Copyright (C) Treasure Data​[2017/07/06 22:33:12] [ info] [engine] started[0] dummy.data: [1499347993.001371317, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}][1] dummy.data: [1499347994.001303118, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}][2] dummy.data: [1499347995.001296133, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}][3] dummy.data: [1499347996.001320284, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]