Output Formats
Some output plugins have a format configuration parameter. Supported values vary by plugin:
HTTP
gelf, json, json_lines, json_stream, msgpack
Kafka
avro (build-dependent), gelf, json, msgpack, raw
Standard Output
json, json_lines, json_stream, msgpack
TCP/TLS
json, json_lines, json_stream, msgpack
UDP
json, json_lines, json_stream, msgpack
WebSocket
gelf, json, json_lines, json_stream, msgpack
Some plugins, such as Kafka, support additional formats not listed here (for example, avro and raw). Always check the individual plugin documentation for the full list of supported values.
The format specifications and references to external documentation are available in this page.
JSON formats
The most custom format options are the JSON types. The following are common variants of sending JSON encoded data over various APIs.
The following example payloads are modified for readability.
Records shown in example payloads are the three first commits of the Fluent Bit repository.
Nesting depth limit
In Fluent Bit version 5.1.3 and greater, converting a record to any of these JSON formats serializes at most 512 levels of nested maps and arrays. Maps and arrays through level 512 are preserved. When a record nests deeper than this limit, Fluent Bit writes each non-empty map or array at level 513 as null, omits its contents, and keeps the rest of the record intact. An empty map or array at level 513 is still written as {} or [].
Fluent Bit logs the following warning at most once for each serialization pass, no matter how many branches are affected:
[warn] [pack] msgpack to JSON conversion exceeded the maximum nesting depth (512), truncating remaining structureWith json, a pass covers the full array of queued records. With json_lines and json_stream, a pass covers one record. If the output buffer is too small, Fluent Bit grows it and retries in a new pass, which can log the warning again.
json
Queued records encoded as a single JSON array of JSON encoded Fluent Bit records.
Example payload
json_stream
a.k.a. Concatenated JSON
Queued JSON-encoded Fluent Bit records, with no separator between records.
Example json_stream payload
json_lines
Also known as newline-delimited JSON.
Queued JSON-encoded Fluent Bit records, separated by line-breaks (\n).
Example json_lines payload
msgpack
MessagePack is a binary format for serializing objects.
Refer to the official MessagePack format specification.
GELF
Graylog Extended Log Format (GELF) is a JSON based log format which is designed to avoid the shortcomings of classic plain syslog.
See the official documentation for the GELF format.
It uses a fixed set of keys, with the addition of adding custom keys prefixed by one single underscore character.
Example GELF payload
Here is an example payload, copied directly from the official documentation.
Last updated
Was this helpful?