tail -f
shell command.Path
pattern and for every new line found (separated by a ), it generates a new record. Optionally a database file can be used so the plugin can have a history of tracked files and a state of offsets, this is very useful to resume a state if the service is restarted.Exclude_Path *.gz,*.zip
normal
mode, but if you really need full synchronization after every write operation you should set full
mode. Note that full
has a high I/O performance cost.DB
is not specified, by default the plugin will start reading each target file from the beginning. This also might cause some unwanted behavior, for example when a line is bigger that Buffer_Chunk_Size
and Skip_Long_Lines
is not turned on, the file will be read from the beginning of each Refresh_Interval
until the file is rotated.multiline.parser
definition, you must disable the old configuration from your tail section like:docker
and cri
multiline formats.parsers.conf
file.Multiline On
in the configuration and use the Parser_Firstline
and additional parser parameters Parser_N
if needed. If we are trying to read the following Java Stacktrace as a single eventParser_Firstline
parameter that matches the first line of a multi-line event. Once a match is made Fluent Bit will read all future lines until another match with Parser_Firstline
is made .time
and the remaining portion of the multiline as log
Parser_N
where N is an integer. The final Fluent Bit configuration looks like the following:Make sure to explore when Fluent Bit is not hard working on the database file, otherwise you will see some Error: database is locked messages.
Write Ahead Log
or WAL
. This allows to improve performance of read and write operations to disk. When enabled, you will see in your file system additional files being created, consider the following configuration statement:test.db
and in the same path for that file SQLite will create two additional files:WAL
mechanism that helps to improve performance and reduce the number system calls required. The -wal
file refers to the file that stores the new changes to be committed, at some point the WAL
file transactions are moved back to the real database file. The -shm
file is a shared-memory type to allow concurrent-users to the WAL
file.WAL
mechanism give us higher performance but also might increase the memory usage by Fluent Bit. Most of this usage comes from the memory mapped and cached pages. In some cases you might see that memory usage keeps a bit high giving the impression of a memory leak, but actually is not relevant unless you want your memory metrics back to normal. Starting from Fluent Bit v1.7.3 we introduced the new option db.journal_mode
mode that sets the journal mode for databases, by default it will be WAL (Write-Ahead Logging)
, currently allowed configurations for db.journal_mode
are DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF
.Path
patterns cannot match the rotated files. Otherwise, the rotated file would be read again and lead to duplicate records.