# Decoders There are cases where the log messages you want to parse contain encoded data. A typical use case can be found in containerized environments with Docker. Docker logs its data in JSON format, which uses escaped strings. Consider the following message generated by the application: ``` {"status": "up and running"} ``` The Docker log message encapsulates something like this: ``` {"log":"{\"status\": \"up and running\"}\r\n","stream":"stdout","time":"2018-03-09T01:01:44.851160855Z"} ``` The original message is handled as an escaped string. Fluent Bit will use the original structured message, and not a string. ## Get started Decoders are a built-in feature of parsers in Fluent Bit. Each parser definition can optionally set one or more decoders. Select from one of these decoder types: * `Decode_Field`: If the content can be decoded in a structured message, append the structured message (keys and values) to the original log message. * `Decode_Field_As`: Any decoded content (unstructured or structured) will be replaced in the same key/value, and no extra keys are added. For example, the predefined Docker parser has the following definition: {% tabs %} {% tab title="parsers.yaml" %} ```yaml parsers: - name: docker format: json time_key: time time_format: '%Y-%m-%dT%H:%M:%S.%L' time_keep: on # Command | Decoder | Field | Optional Action | # ==========|==========|=======|===================| decode_field_as: escaped log ``` {% endtab %} {% tab title="parsers.conf" %} ``` [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On # Command | Decoder | Field | Optional Action | # ==============|===========|=======|===================| Decode_Field_As escaped log ``` {% endtab %} {% endtabs %} Each line in the parser with a key `Decode_Field` instructs the parser to apply a specific decoder on a given field. Optionally, it offers the option to take an extra action if the decoder doesn't succeed. ### Decoder options | Name | Description | | -------------- | ----------------------------------------------------------------------------------------------------------------------- | | `json` | Handle the field content as a JSON map. If the decoder finds a JSON map, it replaces the content with a structured map. | | `escaped` | Decode an escaped string. | | `escaped_utf8` | Decode a UTF8 escaped string. | ### Optional actions If a decoder fails to decode the field, or if you want to try another decoder, you can define an optional action. Available actions are: | Name | Description | | ---------- | ------------------------------------------------------------------------------------------ | | `try_next` | If the decoder failed, apply the next decoder in the list for the same field. | | `do_next` | If the decoder succeeded or failed, apply the next decoder in the list for the same field. | Actions are affected by some restrictions: * `Decode_Field_As`: If successful, another decoder of the same type and the same field can be applied only if the data continues being an unstructured message (raw text). * `Decode_Field`: If successful, can be applied only once for the same field. `Decode_Field` is intended to decode a structured message. ### Examples #### `escaped_utf8` Example input from `/path/to/log.log`: ``` {"log":"\u0009Checking indexes...\n","stream":"stdout","time":"2018-02-19T23:25:29.1845444Z"} {"log":"\u0009\u0009Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary\n","stream":"stdout","time":"2018-02-19T23:25:29.1845536Z"} {"log":"\u0009Done\n","stream":"stdout","time":"2018-02-19T23:25:29.1845622Z"} ``` Example output: ``` ... [24] tail.0: [1519082729.184544400, {"log"=>" Checking indexes... ", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845444Z"}] [25] tail.0: [1519082729.184553600, {"log"=>" Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary ", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845536Z"}] [26] tail.0: [1519082729.184562200, {"log"=>" Done ", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845622Z"}] ... ``` Decoder example Fluent Bit configuration files: {% tabs %} {% tab title="fluent-bit.yaml" %} ```yaml service: parsers_file: parsers.yaml pipeline: inputs: - name: tail parser: docker path: /path/to/log.log outputs: - name: stdout match: '*' ``` {% endtab %} {% tab title="fluent-bit.conf" %} ``` [SERVICE] Parsers_File parsers.conf [INPUT] Name tail Parser docker Path /path/to/log.log [OUTPUT] Name stdout Match * ``` {% endtab %} {% endtabs %} The example parsers file: {% tabs %} {% tab title="parsers.yaml" %} ```yaml parsers: - name: docker format: json time_key: time time_format: '%Y-%m-%dT%H:%M:%S %z' decode_field_as: escaped_utf8 log ``` {% endtab %} {% tab title="parsers.conf" %} ``` [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S %z Decode_Field_as escaped_utf8 log ``` {% endtab %} {% endtabs %} --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.fluentbit.io/manual/4.0/data-pipeline/parsers/decoders.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.