# Decoders

There are certain cases where the log messages being parsed contains encoded data, a typical use case can be found in containerized environments with Docker: application logs it data in JSON format but becomes an escaped string, Consider the following example

Original message generated by the application:

```
{"status": "up and running"}
```

Then the Docker log message become encapsulated as follows:

```
{"log":"{\"status\": \"up and running\"}\r\n","stream":"stdout","time":"2018-03-09T01:01:44.851160855Z"}
```

as you can see the original message is handled as an escaped string. Ideally in Fluent Bit we would like to keep having the original structured message and not a string.

## Getting Started

Decoders are a built-in feature available through the Parsers file, each Parser definition can optionally set one or multiple decoders. There are two type of decoders type:

* Decode\_Field: if the content can be decoded in a structured message, append that structure message (keys and values) to the original log message.
* Decode\_Field\_As: any content decoded (unstructured or structured) will be replaced in the same key/value, no extra keys are added.

Our pre-defined Docker Parser have the following definition:

```
[PARSER]
    Name         docker
    Format       json
    Time_Key     time
    Time_Format  %Y-%m-%dT%H:%M:%S.%L
    Time_Keep    On
    # Command       |  Decoder  | Field | Optional Action   |
    # ==============|===========|=======|===================|
    Decode_Field_As    escaped     log
```

Each line in the parser with a key *Decode\_Field* instruct the parser to apply a specific decoder on a given field, optionally it offer the option to take an extra action if the decoder cannot succeed.

### Decoders

| Name          | Description                                                                                                      |
| ------------- | ---------------------------------------------------------------------------------------------------------------- |
| json          | handle the field content as a JSON map. If it find a JSON map it will replace the content with a structured map. |
| escaped       | decode an escaped string.                                                                                        |
| escaped\_utf8 | decode a UTF8 escaped string.                                                                                    |

### Optional Actions

By default if a decoder fails to decode the field or want to try a next decoder, is possible to define an optional action. Available actions are:

| Name      | Description                                                                                |
| --------- | ------------------------------------------------------------------------------------------ |
| try\_next | if the decoder failed, apply the next Decoder in the list for the same field.              |
| do\_next  | if the decoder succeeded or failed, apply the next Decoder in the list for the same field. |

Note that actions are affected by some restrictions:

* on Decode\_Field\_As, if succeeded, another decoder of the same type in the same field can be applied only if the data continue being a unstructed message (raw text).
* on Decode\_Field, if succeeded, can only be applied once for the same field. By nature Decode\_Field aims to decode a structured message.

### Examples

### escaped\_utf8

Example input (from `/path/to/log.log` in configuration below)

```
{"log":"\u0009Checking indexes...\n","stream":"stdout","time":"2018-02-19T23:25:29.1845444Z"}
{"log":"\u0009\u0009Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary\n","stream":"stdout","time":"2018-02-19T23:25:29.1845536Z"}
{"log":"\u0009Done\n","stream":"stdout","time":"2018-02-19T23:25:29.1845622Z"}
```

Example output

```
[24] tail.0: [1519082729.184544400, {"log"=>"   Checking indexes...                                                   
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845444Z"}]
[25] tail.0: [1519082729.184553600, {"log"=>"           Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845536Z"}]
[26] tail.0: [1519082729.184562200, {"log"=>"   Done                  
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845622Z"}]
```

Configuration file

```
[SERVICE]
    Parsers_File fluent-bit-parsers.conf

[INPUT]
    Name        tail
    Parser      docker
    Path        /path/to/log.log

[OUTPUT]
    Name   stdout
    Match  *
```

The `fluent-bit-parsers.conf` file,

```
[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S %z
    Decode_Field_as escaped_utf8 log
```