The Regex parser lets you define a custom Ruby regular expression that uses a named capture feature to define which content belongs to which key name.
Use Tail Multiline when you need to support regexes across multiple lines from a tail. The Tail input plugin treats each line as a separate entity.
Security Warning: Onigmo is a backtracking regex engine. When using expensive regex patterns Onigmo can take a long time to perform pattern matching. Read "ReDoS" on OWASP for additional information.
Setting the format to regex requires a regex configuration key.
The regex parser supports the following configuration parameters:
Skip_Empty_Values
If enabled, the parser ignores empty value of the record.
True
Fluent Bit uses the Onigmo regular expression library on Ruby mode.
You can use only alphanumeric characters and underscore in group names. For example, a group name like (?<user-name>.*) causes an error due to the invalid dash (-) character. Use the Rubular web editor to test your expressions.
The following parser configuration example provides rules that can be applied to an Apache HTTP Server log entry:
[PARSER]
Name apache
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
Types code:integer size:integerAs an example, review the following Apache HTTP Server log entry:
192.168.2.20 - - [29/Jul/2015:10:27:10 -0300] "GET /cgi-bin/try/ HTTP/1.0" 200 3395This log entry doesn't provide a defined structure for Fluent Bit. Enabling the proper parser can help to make a structured representation of the entry:
[1154104030, {"host"=>"192.168.2.20",
"user"=>"-",
"method"=>"GET",
"path"=>"/cgi-bin/try/",
"code"=>"200",
"size"=>"3395",
"referer"=>"",
"agent"=>""
}
]The ltsv parser allows to parse LTSV formatted texts.
Labeled Tab-separated Values (LTSV format is a variant of Tab-separated Values (TSV). Each record in a LTSV file is represented as a single line. Each field is separated by TAB and has a label and a value. The label and the value have been separated by ':'.
Here is an example how to use this format in the apache access log.
Config this in httpd.conf:
LogFormat "host:%h\tident:%l\tuser:%u\ttime:%t\treq:%r\tstatus:%>s\tsize:%b\treferer:%{Referer}i\tua:%{User-Agent}i" combined_ltsv
CustomLog "logs/access_log" combined_ltsvThe parser.conf:
[PARSER]
Name access_log_ltsv
Format ltsv
Time_Key time
Time_Format [%d/%b/%Y:%H:%M:%S %z]
Types status:integer size:integerThe following log entry is a valid content for the parser defined above:
host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET / HTTP/1.1 status:200 size:16218 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET /assets/plugins/bootstrap/css/bootstrap.min.css HTTP/1.1 status:200 size:121200 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET /assets/css/headers/header-v6.css HTTP/1.1 status:200 size:37706 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET /assets/css/style.css HTTP/1.1 status:200 size:1279 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0After processing, it internal representation will be:
[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET / HTTP/1.1", "status"=>200, "size"=>16218, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}]
[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET /assets/plugins/bootstrap/css/bootstrap.min.css HTTP/1.1", "status"=>200, "size"=>121200, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}]
[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET /assets/css/headers/header-v6.css HTTP/1.1", "status"=>200, "size"=>37706, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}]
[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET /assets/css/style.css HTTP/1.1", "status"=>200, "size"=>1279, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}]The time has been converted to Unix timestamp (UTC).
Parsers are an important component of , with them you can take any unstructured log entry and give them a structure that makes easier it processing and further filtering.
The parser engine is fully configurable and can process log entries based in two types of format:
(named capture)
By default, Fluent Bit provides a set of pre-configured parsers that can be used for different use cases such as logs from:
Apache
Nginx
Docker
Syslog rfc5424
Syslog rfc3164
Parsers are defined in one or multiple configuration files that are loaded at start time, either from the command line or through the main Fluent Bit configuration file.
Note: If you are using Regular Expressions note that Fluent Bit uses Ruby based regular expressions and we encourage to use web site as an online editor to test them.
Multiple parsers can be defined and each section has it own properties. The following table describes the available options for each parser definition:
All parsers must be defined in a parsers.conf file, not in the Fluent Bit global configuration file. The parsers file expose all parsers available that can be used by the Input plugins that are aware of this feature. A parsers file can have multiple entries like this:
For more information about the parsers available, please refer to the default parsers file distributed with Fluent Bit source code:
Time resolution and its format supported are handled by using the libc system function.
In addition, we extended our time resolution to support fractional seconds like 2017-05-17T15:44:31**.187512963**Z. Since Fluent Bit v0.12 we have full support for nanoseconds resolution, the %L format option for Time_Format is provided as a way to indicate that content must be interpreted as fractional seconds.
Note: The option %L is only valid when used after seconds (
%S) or seconds since the Epoch (%s), e.g:%S.%Lor%s.%L
The JSON parser is the simplest option: if the original log source is a JSON map string, it will take its structure and convert it directly to the internal binary representation.
A simple configuration that can be found in the default parsers configuration file, is the entry to parse Docker log files (when the tail input plugin is used):
The following log entry is a valid content for the parser defined above:
After processing, its internal representation will be:
The time has been converted to Unix timestamp (UTC) and the map reduced to each component of the original message.
The logfmt parser allows to parse the logfmt format described in . A more formal description is in .
Here is an example configuration:
The following log entry is a valid content for the parser defined above:
After processing, it internal representation will be:
If you want to be more strict than the logfmt standard and not parse lines where some attributes do not have values (such as key3) in the example above, you can configure the parser as follows:
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S %z{"key1": 12345, "key2": "abc", "time": "2006-07-28T13:22:04Z"}[1154103724, {"key1"=>12345, "key2"=>"abc"}]Name
Set an unique name for the parser in question.
Format
Regex
If format is regex, this option must be set specifying the Ruby Regular Expression that will be used to parse and compose the structured message.
Time_Key
If the log entry provides a field with a timestamp, this option specifies the name of that field.
Time_Format
Specify the format of the time field so it can be recognized and analyzed properly. Fluent-bit uses strptime(3) to parse time so you can refer to strptime documentation for available modifiers.
Time_Offset
Specify a fixed UTC time offset (e.g. -0600, +0200, etc.) for local dates.
Time_Keep
By default when a time key is recognized and parsed, the parser will drop the original time field. Enabling this option will make the parser to keep the original time field and it value in the log entry.
Time_System_Timezone
If there is no timezone (%z) specified in the given Time_Format, enabling this option will make the parser detect and use the system's configured timezone. The configured timezone is detected from the TZ environment variable.
Types
Specify the data type of parsed field. The syntax is types <field_name_1>:<type_name_1> <field_name_2>:<type_name_2> .... The supported types are string(default), integer, bool, float, hex. The option is supported by ltsv, logfmt and regex.
Decode_Field
Decode a field value, the only decoder available is json. The syntax is: Decode_Field json <field_name>.
Skip_Empty_Values
Specify a boolean which determines if the parser should skip empty values. The default is true.
Time_Strict
The default value (true) tells the parser to be strict with the expected time format. With this option set to false, the parser will be permissive with the format of the time. This is useful when the format expects time fraction but the time to be parsed doesn't include it.
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name syslog-rfc5424
Format regex
Regex ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
Types pid:integer[PARSER]
Name logfmt
Format logfmtkey1=val1 key2=val2 key3[1540936693, {"key1"=>"val1",
"key2"=>"val2"
"key3"=>true}][PARSER]
Name logfmt
Format logfmt
Logfmt_No_Bare_Keys trueThere are cases where the log messages being parsed contain encoded data. A typical use case can be found in containerized environments with Docker. Docker logs its data in JSON format, which uses escaped strings.
Consider the following message generated by the application:
The Docker log message encapsulates something like this:
The original message is handled as an escaped string. Fluent Bit wants to use the original structured message and not a string.
Decoders are a built-in feature available through the Parsers file. Each parser definition can optionally set one or more decoders. There are two types of decoders:
Decode_Field: If the content can be decoded in a structured message, append the structured message (keys and values) to the original log message.
Decode_Field_As: Any decoded content (unstructured or structured) will be replaced in the same key/value, and no extra keys are added.
Our pre-defined Docker parser has the following definition:
Each line in the parser with a key Decode_Field instructs the parser to apply a specific decoder on a given field. Optionally, it offers the option to take an extra action if the decoder doesn't succeed.
If a decoder fails to decode the field or, you want to try another decoder, you can define an optional action. Available actions are:
Actions are affected by some restrictions:
Decode_Field_As: If successful, another decoder of the same type and the same field can be applied only if the data continues being an unstructured message (raw text).
Decode_Field: If successful, can only be applied once for the same field. Decode_Field` is intended to decode a structured message.
escaped_utf8Example input from /path/to/log.log:
Example output:
Decoder configuration file:
The fluent-bit-parsers.conf file:
{"status": "up and running"}{"log":"{\"status\": \"up and running\"}\r\n","stream":"stdout","time":"2018-03-09T01:01:44.851160855Z"}[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
# Command | Decoder | Field | Optional Action |
# ==============|===========|=======|===================|
Decode_Field_As escaped logjson
Handle the field content as a JSON map. If it finds a JSON map, it replaces the content with a structured map.
escaped
Decode an escaped string.
escaped_utf8
Decode a UTF8 escaped string.
try_next
if the decoder failed, apply the next decoder in the list for the same field.
do_next
if the decoder succeeded or failed, apply the next decoder in the list for the same field.
{"log":"\u0009Checking indexes...\n","stream":"stdout","time":"2018-02-19T23:25:29.1845444Z"}
{"log":"\u0009\u0009Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary\n","stream":"stdout","time":"2018-02-19T23:25:29.1845536Z"}
{"log":"\u0009Done\n","stream":"stdout","time":"2018-02-19T23:25:29.1845622Z"}[24] tail.0: [1519082729.184544400, {"log"=>" Checking indexes...
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845444Z"}]
[25] tail.0: [1519082729.184553600, {"log"=>" Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845536Z"}]
[26] tail.0: [1519082729.184562200, {"log"=>" Done
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845622Z"}][SERVICE]
Parsers_File fluent-bit-parsers.conf
[INPUT]
Name tail
Parser docker
Path /path/to/log.log
[OUTPUT]
Name stdout
Match *[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S %z
Decode_Field_as escaped_utf8 log