Multiline

The Multiline Filter helps to concatenate messages that originally belong to one context but were split across multiple records or log lines. Common examples are stack traces or applications that print logs in multiple lines.
As part of the built-in functionality, without major configuration effort, you can enable one of ours built-in parsers with auto detection and multi format support:
go
python
ruby
java (Google Cloud Platform Java stacktrace format)
Some comments about this filter:
The usage of this filter depends on a previous configuration of a Multiline Parser definition.
If you wish to concatenate messages read from a log file, it is highly recommended to use the multiline support in the Tail plugin itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages originally split by Docker or CRI container engines, is supported in the Tail plugin.
This filter only performs buffering that persists across different Chunks when Buffer is enabled. Otherwise, the filter will process one Chunk at a time and is not suitable for most inputs which might send multiline messages in separate chunks.
When buffering is enabled, the filter does not immediately emit messages it receives. It uses the in_emitter plugin, same as the Rewrite Tag Filter, and emits messages once they are fully concatenated, or a timeout is reached.
Since concatenated records are re-emitted to the head of the Fluent Bit log pipeline, you can not configure multiple multiline filter definitions that match the same tags. This will cause an infinite loop in the Fluent Bit pipeline; to use multiple parsers on the same logs, configure a single filter definitions with a comma separated list of parsers for multiline.parser. For more, see issue #5235.
Secondly, for the same reason, the multiline filter should be the first filter. Logs will be re-emitted by the multiline filter to the head of the pipeline- the filter will ignore its own re-emitted records, but other filters won't. If there are filters before the multiline filter, they will be applied twice.
Configuration Parameters
The plugin supports the following configuration parameters:
Property
Description
multiline.parser
Specify one or multiple Multiline Parser definitions to apply to the content. You can specify multiple multiline parsers to detect different formats by separating them with a comma.
multiline.key_content
Key name that holds the content to process. Note that a Multiline Parser definition can already specify the key_content to use, but this option allows to overwrite that value for the purpose of the filter.
mode
Mode can be parser for regex concat, or partial_message to concat split docker logs.
buffer
Enable buffered mode. In buffered mode, the filter can concatenate multilines from inputs that ingest records one by one (ex: Forward), rather than in chunks, re-emitting them into the beggining of the pipeline (with the same tag) using the in_emitter instance. With buffer off, this filter will not work with most inputs, except tail.
flush_ms
Flush time for pending multiline records. Defaults to 2000.
emitter_name
Name for the emitter input instance which re-emits the completed records at the beginning of the pipeline.
emitter_storage.type
The storage type for the emitter input instance. This option supports the values memory (default) and filesystem.
emitter_mem_buf_limit
Set a limit on the amount of memory the emitter can consume if the outputs provide backpressure. The default for this limit is 10M. The pipeline will pause once the buffer exceeds the value of this setting. For example, if the value is set to 10M then the pipeline will pause if the buffer exceeds 10M. The pipeline will remain paused until the output drains the buffer below the 10M limit.
Configuration Example
The following example aims to parse a log file called test.log that contains some full lines, a custom Java stacktrace and a Go stacktrace.
The following example files can be located at:

https://github.com/fluent/fluent-bit/tree/master/documentation/examples/multiline/filter_multiline​
Example files content:
fluent-bit.conf
parsers_multiline.conf
test.log
This is the primary Fluent Bit configuration file. It includes the parsers_multiline.conf and tails the file test.log by applying the multiline parsers multiline-regex-test and go. Then it sends the processing to the standard output.
1
[SERVICE]
2
    flush                 1
3
    log_level             info
4
    parsers_file          parsers_multiline.conf
5
​
6
[INPUT]
7
    name                  tail
8
    path                  test.log
9
    read_from_head        true
10
​
11
[FILTER]
12
    name                  multiline
13
    match                 *
14
    multiline.key_content log
15
    multiline.parser      go, multiline-regex-test
16
​
17
[OUTPUT]
18
    name                  stdout
19
    match                 *
20
    
Copied!
This second file defines a multiline parser for the example. Note that a second multiline parser called go is used in fluent-bit.conf, but this one is a built-in parser.
1
[MULTILINE_PARSER]
2
    name          multiline-regex-test
3
    type          regex
4
    flush_timeout 1000
5
    #
6
    # Regex rules for multiline parsing
7
    # ---------------------------------
8
    #
9
    # configuration hints:
10
    #
11
    #  - first state always has the name: start_state
12
    #  - every field in the rule must be inside double quotes
13
    #
14
    # rules |   state name  | regex pattern                  | next state
15
    # ------|---------------|--------------------------------------------
16
    rule      "start_state"   "/(Dec \d+ \d+\:\d+\:\d+)(.*)/"  "cont"
17
    rule      "cont"          "/^\s+at.*/"                     "cont"
18
    
Copied!
An example file with multiline and multiformat content:
1
single line...
2
Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
3
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)
4
    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
5
    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
6
    at com.myproject.module.MyProject.someMethod(MyProject.java:10)
7
    at com.myproject.module.MyProject.main(MyProject.java:6)
8
another line...
9
panic: my panic
10
​
11
goroutine 4 [running]:
12
panic(0x45cb40, 0x47ad70)
13
  /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
14
main.main.func1(0xc420024120)
15
  foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
16
runtime.goexit()
17
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
18
created by main.main
19
  foo.go:5 +0x58
20
​
21
goroutine 1 [chan receive]:
22
runtime.gopark(0x4739b8, 0xc420024178, 0x46fcd7, 0xc, 0xc420028e17, 0x3)
23
  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc420053e30 sp=0xc420053e00 pc=0x42503c
24
runtime.goparkunlock(0xc420024178, 0x46fcd7, 0xc, 0x1000f010040c217, 0x3)
25
  /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc420053e70 sp=0xc420053e30 pc=0x42512e
26
runtime.chanrecv(0xc420024120, 0x0, 0xc420053f01, 0x4512d8)
27
  /usr/local/go/src/runtime/chan.go:506 +0x304 fp=0xc420053f20 sp=0xc420053e70 pc=0x4046b4
28
runtime.chanrecv1(0xc420024120, 0x0)
29
  /usr/local/go/src/runtime/chan.go:388 +0x2b fp=0xc420053f50 sp=0xc420053f20 pc=0x40439b
30
main.main()
31
  foo.go:9 +0x6f fp=0xc420053f80 sp=0xc420053f50 pc=0x4512ef
32
runtime.main()
33
  /usr/local/go/src/runtime/proc.go:185 +0x20d fp=0xc420053fe0 sp=0xc420053f80 pc=0x424bad
34
runtime.goexit()
35
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420053fe8 sp=0xc420053fe0 pc=0x44b4d1
36
​
37
goroutine 2 [force gc (idle)]:
38
runtime.gopark(0x4739b8, 0x4ad720, 0x47001e, 0xf, 0x14, 0x1)
39
  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003e768 sp=0xc42003e738 pc=0x42503c
40
runtime.goparkunlock(0x4ad720, 0x47001e, 0xf, 0xc420000114, 0x1)
41
  /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003e7a8 sp=0xc42003e768 pc=0x42512e
42
runtime.forcegchelper()
43
  /usr/local/go/src/runtime/proc.go:238 +0xcc fp=0xc42003e7e0 sp=0xc42003e7a8 pc=0x424e5c
44
runtime.goexit()
45
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003e7e8 sp=0xc42003e7e0 pc=0x44b4d1
46
created by runtime.init.4
47
  /usr/local/go/src/runtime/proc.go:227 +0x35
48
​
49
goroutine 3 [GC sweep wait]:
50
runtime.gopark(0x4739b8, 0x4ad7e0, 0x46fdd2, 0xd, 0x419914, 0x1)
51
  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003ef60 sp=0xc42003ef30 pc=0x42503c
52
runtime.goparkunlock(0x4ad7e0, 0x46fdd2, 0xd, 0x14, 0x1)
53
  /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003efa0 sp=0xc42003ef60 pc=0x42512e
54
runtime.bgsweep(0xc42001e150)
55
  /usr/local/go/src/runtime/mgcsweep.go:52 +0xa3 fp=0xc42003efd8 sp=0xc42003efa0 pc=0x419973
56
runtime.goexit()
57
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003efe0 sp=0xc42003efd8 pc=0x44b4d1
58
created by runtime.gcenable
59
  /usr/local/go/src/runtime/mgc.go:216 +0x58
60
one more line, no multiline
Copied!
By running Fluent Bit with the given configuration file you will obtain:
1
$ fluent-bit -c fluent-bit.conf 
2
​
3
[0] tail.0: [1626736433.143567481, {"log"=>"single line..."}]
4
[1] tail.0: [1626736433.143570538, {"log"=>"Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
5
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)
6
    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
7
    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
8
    at com.myproject.module.MyProject.someMethod(MyProject.java:10)
9
    at com.myproject.module.MyProject.main(MyProject.java:6)"}]
10
[2] tail.0: [1626736433.143572538, {"log"=>"another line..."}]
11
[3] tail.0: [1626736433.143572894, {"log"=>"panic: my panic
12
​
13
goroutine 4 [running]:
14
panic(0x45cb40, 0x47ad70)
15
  /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
16
main.main.func1(0xc420024120)
17
  foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
18
runtime.goexit()
19
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
20
created by main.main
21
  foo.go:5 +0x58
22
​
23
goroutine 1 [chan receive]:
24
runtime.gopark(0x4739b8, 0xc420024178, 0x46fcd7, 0xc, 0xc420028e17, 0x3)
25
  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc420053e30 sp=0xc420053e00 pc=0x42503c
26
runtime.goparkunlock(0xc420024178, 0x46fcd7, 0xc, 0x1000f010040c217, 0x3)
27
  /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc420053e70 sp=0xc420053e30 pc=0x42512e
28
runtime.chanrecv(0xc420024120, 0x0, 0xc420053f01, 0x4512d8)
29
  /usr/local/go/src/runtime/chan.go:506 +0x304 fp=0xc420053f20 sp=0xc420053e70 pc=0x4046b4
30
runtime.chanrecv1(0xc420024120, 0x0)
31
  /usr/local/go/src/runtime/chan.go:388 +0x2b fp=0xc420053f50 sp=0xc420053f20 pc=0x40439b
32
main.main()
33
  foo.go:9 +0x6f fp=0xc420053f80 sp=0xc420053f50 pc=0x4512ef
34
runtime.main()
35
  /usr/local/go/src/runtime/proc.go:185 +0x20d fp=0xc420053fe0 sp=0xc420053f80 pc=0x424bad
36
runtime.goexit()
37
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420053fe8 sp=0xc420053fe0 pc=0x44b4d1
38
​
39
goroutine 2 [force gc (idle)]:
40
runtime.gopark(0x4739b8, 0x4ad720, 0x47001e, 0xf, 0x14, 0x1)
41
  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003e768 sp=0xc42003e738 pc=0x42503c
42
runtime.goparkunlock(0x4ad720, 0x47001e, 0xf, 0xc420000114, 0x1)
43
  /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003e7a8 sp=0xc42003e768 pc=0x42512e
44
runtime.forcegchelper()
45
  /usr/local/go/src/runtime/proc.go:238 +0xcc fp=0xc42003e7e0 sp=0xc42003e7a8 pc=0x424e5c
46
runtime.goexit()
47
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003e7e8 sp=0xc42003e7e0 pc=0x44b4d1
48
created by runtime.init.4
49
  /usr/local/go/src/runtime/proc.go:227 +0x35
50
​
51
goroutine 3 [GC sweep wait]:
52
runtime.gopark(0x4739b8, 0x4ad7e0, 0x46fdd2, 0xd, 0x419914, 0x1)
53
  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003ef60 sp=0xc42003ef30 pc=0x42503c
54
runtime.goparkunlock(0x4ad7e0, 0x46fdd2, 0xd, 0x14, 0x1)
55
  /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003efa0 sp=0xc42003ef60 pc=0x42512e
56
runtime.bgsweep(0xc42001e150)
57
  /usr/local/go/src/runtime/mgcsweep.go:52 +0xa3 fp=0xc42003efd8 sp=0xc42003efa0 pc=0x419973
58
runtime.goexit()
59
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003efe0 sp=0xc42003efd8 pc=0x44b4d1
60
created by runtime.gcenable
61
  /usr/local/go/src/runtime/mgc.go:216 +0x58"}]
62
[4] tail.0: [1626736433.143585473, {"log"=>"one more line, no multiline"}]
Copied!
The lines that did not match a pattern are not considered as part of the multiline message, while the ones that matched the rules were concatenated properly.
Docker Partial Message Use Case
When Fluent Bit is consuming logs from a container runtime, such as docker, these logs will be split above a certain limit, usually 16KB. If your application emits a 100K log line, it will be split into 7 partial messages. If you are using the Fluentd Docker Log Driver to send the logs to Fluent Bit, they might look like this:
1
{"source": "stdout", "log": "... omitted for brevity...", "partial_message": "true", "partial_id": "dc37eb08b4242c41757d4cd995d983d1cdda4589193755a22fcf47a638317da0", "partial_ordinal": "1", "partial_last": "false", "container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig"}]
Copied!
Fluent Bit can re-combine these logs that were split by the runtime and remove the partial message fields. The filter example below is for this use case.
1
[FILTER]
2
     name                  multiline
3
     match                 *
4
     multiline.key_content log
5
     mode                  partial_message
Copied!
The two options for mode are mutually exclusive in the filter. If you set the mode to partial_message then the multiline.parser option is not allowed.
Modify
Nest
Last modified 23d ago
Export as PDF
Copy link
Contents
Configuration Parameters
Configuration Example
Docker Partial Message Use Case