Parser

The Parser Filter plugin allows for parsing fields in event records.
Configuration Parameters
The plugin supports the following configuration parameters:
Key
Description
Default
Key_Name
Specify field name in record to parse.
​
Parser
Specify the parser name to interpret the field. Multiple Parser entries are allowed (one per line).
​
Preserve_Key
Keep original Key_Name field in the parsed result. If false, the field will be removed.
False
Reserve_Data
Keep all other original fields in the parsed result. If false, all other original fields will be removed.
False
Unescape_Key
If the key is an escaped string (e.g: stringify JSON), unescape the string before applying the parser.
False
Getting Started
Configuration File
This is an example of parsing a record {"data":"100 0.5 true This is example"}.
The plugin needs a parser file which defines how to parse each field.
1
[PARSER]
2
    Name dummy_test
3
    Format regex
4
    Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$
Copied!
The path of the parser file should be written in configuration file under the [SERVICE] section.
1
[SERVICE]
2
    Parsers_File /path/to/parsers.conf
3
​
4
[INPUT]
5
    Name dummy
6
    Tag  dummy.data
7
    Dummy {"data":"100 0.5 true This is example"}
8
​
9
[FILTER]
10
    Name parser
11
    Match dummy.*
12
    Key_Name data
13
    Parser dummy_test
14
​
15
[OUTPUT]
16
    Name stdout
17
    Match *
Copied!
The output is
1
$ fluent-bit -c dummy.conf
2
Fluent Bit v1.x.x
3
* Copyright (C) 2019-2020 The Fluent Bit Authors
4
* Copyright (C) 2015-2018 Treasure Data
5
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
6
* https://fluentbit.io
7
​
8
[2017/07/06 22:33:12] [ info] [engine] started
9
[0] dummy.data: [1499347993.001371317, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
10
[1] dummy.data: [1499347994.001303118, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
11
[2] dummy.data: [1499347995.001296133, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
12
[3] dummy.data: [1499347996.001320284, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
Copied!
You can see the records {"data":"100 0.5 true This is example"} are parsed.
Preserve original fields
By default, the parser plugin only keeps the parsed fields in its output.
If you enable Reserve_Data, all other fields are preserved:
1
[PARSER]
2
    Name dummy_test
3
    Format regex
4
    Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$
Copied!
1
[SERVICE]
2
    Parsers_File /path/to/parsers.conf
3
​
4
[INPUT]
5
    Name dummy
6
    Tag  dummy.data
7
    Dummy {"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"}
8
​
9
[FILTER]
10
    Name parser
11
    Match dummy.*
12
    Key_Name data
13
    Parser dummy_test
14
    Reserve_Data On
Copied!
This will produce the output:
1
$ fluent-bit -c dummy.conf
2
Fluent-Bit v0.12.0
3
Copyright (C) Treasure Data
4
​
5
[2017/07/06 22:33:12] [ info] [engine] started
6
[0] dummy.data: [1499347993.001371317, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
7
[1] dummy.data: [1499347994.001303118, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
8
[2] dummy.data: [1499347995.001296133, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
9
[3] dummy.data: [1499347996.001320284, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
Copied!
If you enable Reserved_Data and Preserve_Key, the original key field will be preserved as well:
1
[PARSER]
2
    Name dummy_test
3
    Format regex
4
    Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$
Copied!
1
[SERVICE]
2
    Parsers_File /path/to/parsers.conf
3
​
4
[INPUT]
5
    Name dummy
6
    Tag  dummy.data
7
    Dummy {"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"}
8
​
9
[FILTER]
10
    Name parser
11
    Match dummy.*
12
    Key_Name data
13
    Parser dummy_test
14
    Reserve_Data On
15
    Preserve_Key On
Copied!
This will produce the following output:
1
$ fluent-bit -c dummy.conf
2
Fluent-Bit v0.12.0
3
Copyright (C) Treasure Data
4
​
5
[2017/07/06 22:33:12] [ info] [engine] started
6
[0] dummy.data: [1499347993.001371317, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
7
[1] dummy.data: [1499347994.001303118, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
8
[2] dummy.data: [1499347995.001296133, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
9
[3] dummy.data: [1499347996.001320284, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
Copied!

Lua

Record Modifier

Last modified 6mo ago

Export as PDF

Copy link

Contents

Configuration Parameters

Getting Started

Configuration File

Preserve original fields