tail -fshell command.
Pathpattern and for every new line found (separated by a ), it generates a new record. Optionally a database file can be used so the plugin can have a history of tracked files and a state of offsets, this is very useful to resume a state if the service is restarted.
DBis not specified, by default the plugin will start reading each target file from the beginning. This also might cause some unwanted behavior, for example when a line is bigger that
Skip_Long_Linesis not turned on, the file will be read from the beginning of each
Refresh_Intervaluntil the file is rotated.
multiline.parserdefinition, you must disable the old configuration from your tail section like:
Multiline Onin the configuration and use the
Parser_Firstlineand additional parser parameters
Parser_Nif needed. If we are trying to read the following Java Stacktrace as a single event
Parser_Firstlineparameter that matches the first line of a multi-line event. Once a match is made Fluent Bit will read all future lines until another match with
Parser_Firstlineis made .
timeand the remaining portion of the multiline as
Parser_Nwhere N is an integer. The final Fluent Bit configuration looks like the following:
Make sure to explore when Fluent Bit is not hard working on the database file, otherwise you will see some Error: database is locked messages.
Write Ahead Logor
WAL. This allows to improve performance of read and write operations to disk. When enabled, you will see in your file system additional files being created, consider the following configuration statement:
test.dband in the same path for that file SQLite will create two additional files:
WALmechanism that helps to improve performance and reduce the number system calls required. The
-walfile refers to the file that stores the new changes to be committed, at some point the
WALfile transactions are moved back to the real database file. The
-shmfile is a shared-memory type to allow concurrent-users to the
WALmechanism give us higher performance but also might increase the memory usage by Fluent Bit. Most of this usage comes from the memory mapped and cached pages. In some cases you might see that memory usage keeps a bit high giving the impression of a memory leak, but actually is not relevant unless you want your memory metrics back to normal. Starting from Fluent Bit v1.7.3 we introduced the new option
db.journal_modemode that sets the journal mode for databases, by default it will be
WAL (Write-Ahead Logging), currently allowed configurations for
DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF.
Pathpatterns cannot match the rotated files. Otherwise, the rotated file would be read again and lead to duplicate records.