# Decoders

There are cases where the log messages you want to parse contain encoded data. A typical use case can be found in containerized environments with Docker. Docker logs its data in JSON format, which uses escaped strings.

Consider the following message generated by the application:

```
{"status": "up and running"}
```

The Docker log message encapsulates something like this:

```
{"log":"{\"status\": \"up and running\"}\r\n","stream":"stdout","time":"2018-03-09T01:01:44.851160855Z"}
```

The original message is handled as an escaped string. Fluent Bit will use the original structured message, and not a string.

## Get started

Decoders are a built-in feature of parsers in Fluent Bit. Each parser definition can optionally set one or more decoders. Select from one of these decoder types:

* `Decode_Field`: If the content can be decoded in a structured message, append the structured message (keys and values) to the original log message.
* `Decode_Field_As`: Any decoded content (unstructured or structured) will be replaced in the same key/value, and no extra keys are added.

For example, the predefined Docker parser has the following definition:

{% tabs %}
{% tab title="parsers.yaml" %}

```yaml
parsers:
  - name: docker
    format: json
    time_key: time
    time_format: '%Y-%m-%dT%H:%M:%S.%L'
    time_keep: on
    # Command   |  Decoder | Field | Optional Action   |
    # ==========|==========|=======|===================|
    decode_field_as: escaped log
```

{% endtab %}

{% tab title="parsers.conf" %}

```
[PARSER]
  Name         docker
  Format       json
  Time_Key     time
  Time_Format  %Y-%m-%dT%H:%M:%S.%L
  Time_Keep    On
  # Command       |  Decoder  | Field | Optional Action   |
  # ==============|===========|=======|===================|
  Decode_Field_As    escaped     log
```

{% endtab %}
{% endtabs %}

Each line in the parser with a key `Decode_Field` instructs the parser to apply a specific decoder on a given field. Optionally, it offers the option to take an extra action if the decoder doesn't succeed.

### Decoder options

| Name           | Description                                                                                                             |
| -------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `json`         | Handle the field content as a JSON map. If the decoder finds a JSON map, it replaces the content with a structured map. |
| `escaped`      | Decode an escaped string.                                                                                               |
| `escaped_utf8` | Decode a UTF8 escaped string.                                                                                           |

### Optional actions

If a decoder fails to decode the field, or if you want to try another decoder, you can define an optional action. Available actions are:

| Name       | Description                                                                                |
| ---------- | ------------------------------------------------------------------------------------------ |
| `try_next` | If the decoder failed, apply the next decoder in the list for the same field.              |
| `do_next`  | If the decoder succeeded or failed, apply the next decoder in the list for the same field. |

Actions are affected by some restrictions:

* `Decode_Field_As`: If successful, another decoder of the same type and the same field can be applied only if the data continues being an unstructured message (raw text).
* `Decode_Field`: If successful, can be applied only once for the same field. `Decode_Field` is intended to decode a structured message.

### Examples

#### `escaped_utf8`

Example input from `/path/to/log.log`:

```
{"log":"\u0009Checking indexes...\n","stream":"stdout","time":"2018-02-19T23:25:29.1845444Z"}
{"log":"\u0009\u0009Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary\n","stream":"stdout","time":"2018-02-19T23:25:29.1845536Z"}
{"log":"\u0009Done\n","stream":"stdout","time":"2018-02-19T23:25:29.1845622Z"}
```

Example output:

```
...
[24] tail.0: [1519082729.184544400, {"log"=>"   Checking indexes...
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845444Z"}]
[25] tail.0: [1519082729.184553600, {"log"=>"           Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845536Z"}]
[26] tail.0: [1519082729.184562200, {"log"=>"   Done
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845622Z"}]
...
```

Decoder example Fluent Bit configuration files:

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
service:
  parsers_file: parsers.yaml

pipeline:
  inputs:
    - name: tail
      parser: docker
      path: /path/to/log.log

  outputs:
    - name: stdout
      match: '*'
```

{% endtab %}

{% tab title="fluent-bit.conf" %}

```
[SERVICE]
  Parsers_File parsers.conf

[INPUT]
  Name        tail
  Parser      docker
  Path        /path/to/log.log

[OUTPUT]
  Name   stdout
  Match  *
```

{% endtab %}
{% endtabs %}

The example parsers file:

{% tabs %}
{% tab title="parsers.yaml" %}

```yaml
parsers:
  - name: docker
    format: json
    time_key: time
    time_format: '%Y-%m-%dT%H:%M:%S %z'
    decode_field_as: escaped_utf8 log
```

{% endtab %}

{% tab title="parsers.conf" %}

```
[PARSER]
  Name        docker
  Format      json
  Time_Key    time
  Time_Format %Y-%m-%dT%H:%M:%S %z
  Decode_Field_as escaped_utf8 log
```

{% endtab %}
{% endtabs %}