Decoders
There are cases where the log messages you want to parse contain encoded data. A typical use case can be found in containerized environments with Docker. Docker logs its data in JSON format, which uses escaped strings.
Consider the following message generated by the application:
{"status": "up and running"}
The Docker log message encapsulates something like this:
{"log":"{\"status\": \"up and running\"}\r\n","stream":"stdout","time":"2018-03-09T01:01:44.851160855Z"}
The original message is handled as an escaped string. Fluent Bit will use the original structured message, and not a string.
Get started
Decoders are a built-in feature of parsers in Fluent Bit. Each parser definition can optionally set one or more decoders. Select from one of these decoder types:
Decode_Field
: If the content can be decoded in a structured message, append the structured message (keys and values) to the original log message.Decode_Field_As
: Any decoded content (unstructured or structured) will be replaced in the same key/value, and no extra keys are added.
For example, the predefined Docker parser has the following definition:
parsers:
- name: docker
format: json
time_key: time
time_format: '%Y-%m-%dT%H:%M:%S.%L'
time_keep: on
# Command | Decoder | Field | Optional Action |
# ==========|==========|=======|===================|
decode_field_as: escaped log
Each line in the parser with a key Decode_Field
instructs the parser to apply a specific decoder on a given field. Optionally, it offers the option to take an extra action if the decoder doesn't succeed.
Decoder options
json
Handle the field content as a JSON map. If the decoder finds a JSON map, it replaces the content with a structured map.
escaped
Decode an escaped string.
escaped_utf8
Decode a UTF8 escaped string.
Optional actions
If a decoder fails to decode the field, or if you want to try another decoder, you can define an optional action. Available actions are:
try_next
If the decoder failed, apply the next decoder in the list for the same field.
do_next
If the decoder succeeded or failed, apply the next decoder in the list for the same field.
Actions are affected by some restrictions:
Decode_Field_As
: If successful, another decoder of the same type and the same field can be applied only if the data continues being an unstructured message (raw text).Decode_Field
: If successful, can be applied only once for the same field.Decode_Field
is intended to decode a structured message.
Examples
escaped_utf8
escaped_utf8
Example input from /path/to/log.log
:
{"log":"\u0009Checking indexes...\n","stream":"stdout","time":"2018-02-19T23:25:29.1845444Z"}
{"log":"\u0009\u0009Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary\n","stream":"stdout","time":"2018-02-19T23:25:29.1845536Z"}
{"log":"\u0009Done\n","stream":"stdout","time":"2018-02-19T23:25:29.1845622Z"}
Example output:
[24] tail.0: [1519082729.184544400, {"log"=>" Checking indexes...
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845444Z"}]
[25] tail.0: [1519082729.184553600, {"log"=>" Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845536Z"}]
[26] tail.0: [1519082729.184562200, {"log"=>" Done
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845622Z"}]
Decoder example Fluent Bit configuration files:
service:
parsers_file: parsers.yaml
pipeline:
inputs:
- name: tail
parser: docker
path: /path/to/log.log
outputs:
- name: stdout
match: '*'
The example parsers file:
parsers:
- name: docker
format: json
time_key: time
time_format: '%Y-%m-%dT%H:%M:%S %z'
decode_field_as: escaped_utf8 log
Last updated
Was this helpful?