Multiline
Concatenate multiline or stack trace log messages. Available on Fluent Bit >= v1.8.2.
Last updated
Was this helpful?
Concatenate multiline or stack trace log messages. Available on Fluent Bit >= v1.8.2.
Last updated
Was this helpful?
The Multiline filter helps concatenate messages that originally belonged to one context but were split across multiple records or log lines. Common examples are stack traces or applications that print logs in multiple lines.
Along with multiline filters, you can enable one of the following built-in Fluent Bit parsers with auto detection and multi-format support:
Go
Python
Ruby
Java (Google Cloud Platform Java stack trace format)
When using this filter:
The usage of this filter depends on a previous configuration of a definition.
To concatenate messages read from a log file, it's highly recommended to use the multiline support in the itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages originally split by Docker or CRI container engines, is supported in the .
This filter only performs buffering that persists across different Chunks when Buffer is enabled. Otherwise, the filter processes one chunk at a time and isn't suitable for most inputs which might send multiline messages in separate chunks.
When buffering is enabled, the filter doesn't immediately emit messages it receives. It uses the in_emitter plugin, similar to the , and emits messages once they're fully concatenated, or a timeout is reached.
Since concatenated records are re-emitted to the head of the Fluent Bit log pipeline, you can not configure multiple multiline filter definitions that match the same tags. This will cause an infinite loop in the Fluent Bit pipeline; to use multiple parsers on the same logs, configure a single filter definitions with a comma separated list of parsers for multiline.parser. For more, see issue .
Secondly, for the same reason, the multiline filter should be the first filter. Logs will be re-emitted by the multiline filter to the head of the pipeline- the filter will ignore its own re-emitted records, but other filters won't. If there are filters before the multiline filter, they will be applied twice.
The plugin supports the following configuration parameters:
multiline.parser
multiline.key_content
Key name that holds the content to process. A multiline parser definition can specify the key_content This option allows for overwriting that value for the purpose of the filter.
mode
Mode can be parser for regular expression concatenation, or partial_message to concatenate split Docker logs.
buffer
Enable buffered mode. In buffered mode, the filter can concatenate multiple lines from inputs that ingest records one by one (like Forward), rather than in chunks, re-emitting them into the beginning of the pipeline (with the same tag) using the in_emitter instance. With buffer off, this filter won't work with most inputs, except Tail.
flush_ms
Flush time for pending multiline records. Default: 2000.
emitter_name
Name for the emitter input instance which re-emits the completed records at the beginning of the pipeline.
emitter_storage.type
The storage type for the emitter input instance. This option supports the values memory (default) and filesystem.
emitter_mem_buf_limit
Set a limit on the amount of memory the emitter can consume if the outputs provide backpressure. The default for this limit is 10M. The pipeline will pause once the buffer exceeds the value of this setting. or example, if the value is set to 10M then the pipeline pauses if the buffer exceeds 10M. The pipeline will remain paused until the output drains the buffer below the 10M limit.
The following example aims to parse a log file called test.log that contains some full lines, a custom Java stack trace and a Go Stack Trace.
Example files content:
This is the primary Fluent Bit configuration file. It includes the parsers_multiline.conf and tails the file test.log by applying the multiline parsers multiline-regex-test and go. Then it sends the processing to the standard output.
Running Fluent Bit with the given configuration file:
Should return something like the following:
Lines that don't match a pattern aren't considered as part of the multiline message, while the ones that matched the rules were concatenated properly.
Fluent Bit can re-combine these logs that were split by the runtime and remove the partial message fields. The following filter example is for this use case.
The two options for mode are mutually exclusive in the filter. If you set the mode to partial_message then the multiline.parser option isn't allowed.
Specify one or multiple to apply to the content. You can specify multiple multiline parsers to detect different formats by separating them with a comma.
The following example files can be located .
When Fluent Bit is consuming logs from a container runtime, such as Docker, these logs will be split when larger than a certain limit, usually 16KB. If your application emits a 100K log line, it will be split into seven partial messages. If you are using the to send the logs to Fluent Bit, they might look like this: