Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
The logfmt parser allows to parse the logfmt format described in https://brandur.org/logfmt . A more formal description is in https://godoc.org/github.com/kr/logfmt .
Here is an example configuration:
The following log entry is a valid content for the parser defined above:
After processing, it internal representation will be:
If you want to be more strict than the logfmt standard and not parse lines where some attributes do not have values (such as key3) in the example above, you can configure the parser as follows:
The JSON parser is the simplest option: if the original log source is a JSON map string, it will take its structure and convert it directly to the internal binary representation.
A simple configuration that can be found in the default parsers configuration file, is the entry to parse Docker log files (when the tail input plugin is used):
The following log entry is a valid content for the parser defined above:
After processing, its internal representation will be:
The time has been converted to Unix timestamp (UTC) and the map reduced to each component of the original message.
The ltsv parser allows to parse LTSV formatted texts.
Labeled Tab-separated Values (LTSV format is a variant of Tab-separated Values (TSV). Each record in a LTSV file is represented as a single line. Each field is separated by TAB and has a label and a value. The label and the value have been separated by ':'.
Here is an example how to use this format in the apache access log.
Config this in httpd.conf:
The parser.conf:
The following log entry is a valid content for the parser defined above:
After processing, it internal representation will be:
The time has been converted to Unix timestamp (UTC).
There are cases where the log messages being parsed contain encoded data. A typical use case can be found in containerized environments with Docker. Docker logs its data in JSON format, which uses escaped strings.
Consider the following message generated by the application:
The Docker log message encapsulates something like this:
The original message is handled as an escaped string. Fluent Bit wants to use the original structured message and not a string.
Decoders are a built-in feature available through the Parsers file. Each parser definition can optionally set one or more decoders. There are two types of decoders:
Decode_Field: If the content can be decoded in a structured message, append the structured message (keys and values) to the original log message.
Decode_Field_As: Any decoded content (unstructured or structured) will be replaced in the same key/value, and no extra keys are added.
Our pre-defined Docker parser has the following definition:
Each line in the parser with a key Decode_Field instructs the parser to apply a specific decoder on a given field. Optionally, it offers the option to take an extra action if the decoder doesn't succeed.
If a decoder fails to decode the field or, you want to try another decoder, you can define an optional action. Available actions are:
Actions are affected by some restrictions:
Decode_Field_As: If successful, another decoder of the same type and the same field can be applied only if the data continues being an unstructured message (raw text).
Decode_Field: If successful, can only be applied once for the same field. Decode_Field` is intended to decode a structured message.
escaped_utf8Example input from /path/to/log.log:
Example output:
Decoder configuration file:
The fluent-bit-parsers.conf file:
The Regex parser lets you define a custom Ruby regular expression that uses a named capture feature to define which content belongs to which key name.
Use Tail Multiline when you need to support regexes across multiple lines from a tail. The Tail input plugin treats each line as a separate entity.
Security Warning: Onigmo is a backtracking regex engine. When using expensive regex patterns Onigmo can take a long time to perform pattern matching. Read "ReDoS" on OWASP for additional information.
Setting the format to regex requires a regex configuration key.
The regex parser supports the following configuration parameters:
| Key | Description | Default Value |
|---|---|---|
Fluent Bit uses the Onigmo regular expression library on Ruby mode.
You can use only alphanumeric characters and underscore in group names. For example, a group name like (?<user-name>.*) causes an error due to the invalid dash (-) character. Use the Rubular web editor to test your expressions.
The following parser configuration example provides rules that can be applied to an Apache HTTP Server log entry:
As an example, review the following Apache HTTP Server log entry:
This log entry doesn't provide a defined structure for Fluent Bit. Enabling the proper parser can help to make a structured representation of the entry:
Parsers are an important component of Fluent Bit, with them you can take any unstructured log entry and give them a structure that makes easier it processing and further filtering.
The parser engine is fully configurable and can process log entries based in two types of format:
Regular Expressions (named capture)
By default, Fluent Bit provides a set of pre-configured parsers that can be used for different use cases such as logs from:
Apache
Nginx
Docker
Syslog rfc5424
Syslog rfc3164
Parsers are defined in one or multiple configuration files that are loaded at start time, either from the command line or through the main Fluent Bit configuration file.
Note: If you are using Regular Expressions note that Fluent Bit uses Ruby based regular expressions and we encourage to use Rubular web site as an online editor to test them.
Multiple parsers can be defined and each section has it own properties. The following table describes the available options for each parser definition:
| Key | Description |
|---|---|
All parsers must be defined in a parsers.conf file, not in the Fluent Bit global configuration file. The parsers file expose all parsers available that can be used by the Input plugins that are aware of this feature. A parsers file can have multiple entries like this:
For more information about the parsers available, please refer to the default parsers file distributed with Fluent Bit source code:
https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf
Time resolution and its format supported are handled by using the strftime(3) libc system function.
In addition, we extended our time resolution to support fractional seconds like 2017-05-17T15:44:31**.187512963**Z. Since Fluent Bit v0.12 we have full support for nanoseconds resolution, the %L format option for Time_Format is provided as a way to indicate that content must be interpreted as fractional seconds.
Note: The option %L is only valid when used after seconds (
%S) or seconds since the Epoch (%s), e.g:%S.%Lor%s.%L
| Name | Description |
|---|---|
| Name | Description |
|---|---|
json
Handle the field content as a JSON map. If it finds a JSON map, it replaces the content with a structured map.
escaped
Decode an escaped string.
escaped_utf8
Decode a UTF8 escaped string.
try_next
if the decoder failed, apply the next decoder in the list for the same field.
do_next
if the decoder succeeded or failed, apply the next decoder in the list for the same field.
Skip_Empty_Values
If enabled, the parser ignores empty value of the record.
True
Name
Set an unique name for the parser in question.
Format
Regex
If format is regex, this option must be set specifying the Ruby Regular Expression that will be used to parse and compose the structured message.
Time_Key
If the log entry provides a field with a timestamp, this option specifies the name of that field.
Time_Format
Specify the format of the time field so it can be recognized and analyzed properly. Fluent-bit uses strptime(3) to parse time so you can refer to strptime documentation for available modifiers.
Time_Offset
Specify a fixed UTC time offset (e.g. -0600, +0200, etc.) for local dates.
Time_Keep
By default when a time key is recognized and parsed, the parser will drop the original time field. Enabling this option will make the parser to keep the original time field and it value in the log entry.
Time_System_Timezone
If there is no timezone (%z) specified in the given Time_Format, enabling this option will make the parser detect and use the system's configured timezone. The configured timezone is detected from the TZ environment variable.
Types
Specify the data type of parsed field. The syntax is types <field_name_1>:<type_name_1> <field_name_2>:<type_name_2> .... The supported types are string(default), integer, bool, float, hex. The option is supported by ltsv, logfmt and regex.
Decode_Field
Decode a field value, the only decoder available is json. The syntax is: Decode_Field json <field_name>.
Skip_Empty_Values
Specify a boolean which determines if the parser should skip empty values. The default is true.
Time_Strict
The default value (true) tells the parser to be strict with the expected time format. With this option set to false, the parser will be permissive with the format of the time. This is useful when the format expects time fraction but the time to be parsed doesn't include it.