Decoders
There are certain cases where the log messages being parsed contains encoded data, a typical use case can be found in containerized environments with Docker: application logs it data in JSON format but becomes an escaped string, Consider the following example
Original message generated by the application:
1
{"status": "up and running"}
Copied!
Then the Docker log message become encapsulated as follows:
1
{"log":"{\"status\": \"up and running\"}\r\n","stream":"stdout","time":"2018-03-09T01:01:44.851160855Z"}
Copied!
as you can see the original message is handled as an escaped string. Ideally in Fluent Bit we would like to keep having the original structured message and not a string.

Getting Started

Decoders are a built-in feature available through the Parsers file, each Parser definition can optionally set one or multiple decoders. There are two type of decoders type:
    Decode_Field: if the content can be decoded in a structured message, append that structure message (keys and values) to the original log message.
    Decode_Field_As: any content decoded (unstructured or structured) will be replaced in the same key/value, no extra keys are added.
Our pre-defined Docker Parser have the following definition:
1
[PARSER]
2
Name docker
3
Format json
4
Time_Key time
5
Time_Format %Y-%m-%dT%H:%M:%S.%L
6
Time_Keep On
7
# Command | Decoder | Field | Optional Action |
8
# ==============|===========|=======|===================|
9
Decode_Field_As escaped log
Copied!
Each line in the parser with a key Decode_Field instruct the parser to apply a specific decoder on a given field, optionally it offer the option to take an extra action if the decoder cannot succeed.

Decoders

Name
Description
json
handle the field content as a JSON map. If it find a JSON map it will replace the content with a structured map.
escaped
decode an escaped string.
escaped_utf8
decode a UTF8 escaped string.

Optional Actions

By default if a decoder fails to decode the field or want to try a next decoder, is possible to define an optional action. Available actions are:
Name
Description
try_next
if the decoder failed, apply the next Decoder in the list for the same field.
do_next
if the decoder succeeded or failed, apply the next Decoder in the list for the same field.
Note that actions are affected by some restrictions:
    on Decode_Field_As, if succeeded, another decoder of the same type in the same field can be applied only if the data continues being an unstructured message (raw text).
    on Decode_Field, if succeeded, can only be applied once for the same field. By nature Decode_Field aims to decode a structured message.

Examples

escaped_utf8

Example input (from /path/to/log.log in configuration below)
1
{"log":"\u0009Checking indexes...\n","stream":"stdout","time":"2018-02-19T23:25:29.1845444Z"}
2
{"log":"\u0009\u0009Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary\n","stream":"stdout","time":"2018-02-19T23:25:29.1845536Z"}
3
{"log":"\u0009Done\n","stream":"stdout","time":"2018-02-19T23:25:29.1845622Z"}
Copied!
Example output
1
[24] tail.0: [1519082729.184544400, {"log"=>" Checking indexes...
2
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845444Z"}]
3
[25] tail.0: [1519082729.184553600, {"log"=>" Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary
4
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845536Z"}]
5
[26] tail.0: [1519082729.184562200, {"log"=>" Done
6
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845622Z"}]
Copied!
Configuration file
1
[SERVICE]
2
Parsers_File fluent-bit-parsers.conf
3
4
[INPUT]
5
Name tail
6
Parser docker
7
Path /path/to/log.log
8
9
[OUTPUT]
10
Name stdout
11
Match *
Copied!
The fluent-bit-parsers.conf file,
1
[PARSER]
2
Name docker
3
Format json
4
Time_Key time
5
Time_Format %Y-%m-%dT%H:%M:%S %z
6
Decode_Field_as escaped_utf8 log
Copied!
Last modified 6mo ago