Decoders
There are cases where the log messages being parsed contain encoded data. A typical use case can be found in containerized environments with Docker. Docker logs its data in JSON format, which uses escaped strings.
Consider the following message generated by the application:
The Docker log message encapsulates something like this:
The original message is handled as an escaped string. Fluent Bit wants to use the original structured message and not a string.
Getting Started
Decoders are a built-in feature available through the Parsers file. Each parser definition can optionally set one or more decoders. There are two types of decoders:
Decode_Field
: If the content can be decoded in a structured message, append the structured message (keys and values) to the original log message.Decode_Field_As
: Any decoded content (unstructured or structured) will be replaced in the same key/value, and no extra keys are added.
Our pre-defined Docker parser has the following definition:
Each line in the parser with a key Decode_Field
instructs the parser to apply a specific decoder on a given field. Optionally, it offers the option to take an extra action if the decoder doesn't succeed.
Decoder options
Name | Description |
---|---|
| Handle the field content as a JSON map. If it finds a JSON map, it replaces the content with a structured map. |
| Decode an escaped string. |
| Decode a UTF8 escaped string. |
Optional Actions
If a decoder fails to decode the field or, you want to try another decoder, you can define an optional action. Available actions are:
Name | Description |
---|---|
| if the decoder failed, apply the next decoder in the list for the same field. |
| if the decoder succeeded or failed, apply the next decoder in the list for the same field. |
Actions are affected by some restrictions:
Decode_Field_As
: If successful, another decoder of the same type and the same field can be applied only if the data continues being an unstructured message (raw text).Decode_Field
: If successful, can only be applied once for the same field.Decode
_Field` is intended to decode a structured message.
Examples
escaped_utf8
escaped_utf8
Example input from /path/to/log.log
:
Example output:
Decoder configuration file:
The fluent-bit-parsers.conf
file:
Last updated