Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
High Performance Telemetry Agent for Logs, Metrics and Traces
Fluent Bit is a Fast and Lightweight Telemetry Agent for Logs, Metrics, and Traces for Linux, macOS, Windows, and BSD family operating systems. It has been made with a strong focus on performance to allow the collection and processing of telemetry data from different sources without complexity.
High Performance: High throughput with low resources consumption
Metrics Support: Prometheus and OpenTelemetry compatible
Reliability and Data Integrity
Backpressure Handling
Data Buffering in memory and file system
Networking
Security: built-in TLS/SSL support
Asynchronous I/O
Pluggable Architecture and Extensibility: Inputs, Filters and Outputs
More than 100 built-in plugins are available
Extensibility
Write any input, filter or output plugin in C language
WASM: WASM Filter Plugins or WASM Input Plugins
Bonus: write Filters in Lua or Output plugins in Golang
Monitoring: expose internal metrics over HTTP in JSON and Prometheus format
Stream Processing: Perform data selection and transformation using simple SQL queries
Create new streams of data using query results
Aggregation Windows
Data analysis and prediction: Timeseries forecasting
Portable: runs on Linux, macOS, Windows and BSD systems
Fluent Bit is a CNCF graduated sub-project under the umbrella of Fluentd. Fluent Bit is licensed under the terms of the Apache License v2.0.
Fluent Bit was originally created by Eduardo Silva. As a CNCF-hosted project, it is a fully vendor-neutral and community-driven project.
The way to gather data from your sources
provides different Input Plugins to gather information from different sources, some of them just collect data from log files while others can gather metrics information from the operating system. There are many plugins for different needs.
When an input plugin is loaded, an internal instance is created. Every instance has its own and independent configuration. Configuration keys are often called properties.
Every input plugin has its own documentation section where it's specified how it can be used and what properties are available.
Performance and Data Safety
When processes data, it uses the system memory (heap) as a primary and temporary place to store the record logs before they get delivered, in this private memory area the records are processed.
Buffering refers to the ability to store the records somewhere, and while they are processed and delivered, still be able to store more. Buffering in memory is the fastest mechanism, but there are certain scenarios where it requires special strategies to deal with , data safety or reduce memory consumption by the service in constrained environments.
Network failures or latency on third party service is pretty common, and on scenarios where we cannot deliver data fast enough as we receive new data to process, we likely will face backpressure.
Our buffering strategies are designed to solve problems associated with backpressure and general delivery failures.
Fluent Bit as buffering strategies go, offers a primary buffering mechanism in memory and an optional secondary one using the file system. With this hybrid solution you can accommodate any use case safely and keep a high performance while processing your data.
Both mechanisms are not mutually exclusive and when the data is ready to be processed or delivered it will always be in memory, while other data in the queue might be in the file system until is ready to be processed and moved up to memory.
To learn more about the buffering configuration in Fluent Bit, please jump to the section.
Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
is an open-source telemetry agent specifically designed to efficiently handle the challenges of collecting and processing telemetry data across a wide range of environments, from constrained systems to complex cloud infrastructures. Managing telemetry data from various sources and formats can be a constant challenge, particularly when performance is a critical factor.
Rather than serving as a drop-in replacement, Fluent Bit enhances the observability strategy for your infrastructure by adapting and optimizing your existing logging layer, as well as metrics and traces processing. Furthermore, Fluent Bit supports a vendor-neutral approach, seamlessly integrating with other ecosystems such as Prometheus and OpenTelemetry. Trusted by major cloud providers, banks, and companies in need of a ready-to-use telemetry agent solution, Fluent Bit effectively manages diverse data sources and formats while maintaining optimal performance.
Fluent Bit can be deployed as an edge agent for localized telemetry data handling or utilized as a central aggregator/collector for managing telemetry data across multiple sources and environments.
has been designed with performance and low resource consumption in mind.
For more details, please refer to the section.
Strong Commitment to the Openness and Collaboration
Fluent Bit, including its core, plugins and tools are distributed under the terms of the Apache License v2.0:
The Production Grade Telemetry Ecosystem
Telemetry data processing in general can be complex, and at scale a bit more, that's why Fluentd was born. Fluentd has become more than a simple tool, it has grown into a fullscale ecosystem that contains SDKs for different languages and sub-projects like Fluent Bit.
On this page, we will describe the relationship between the Fluentd and Fluent Bit open source projects, as a summary we can say both are:
Licensed under the terms of Apache License v2.0
Graduated Hosted projects by the Cloud Native Computing Foundation (CNCF)
Production Grade solutions: deployed million of times every single day.
Vendor neutral and community driven projects
Widely Adopted by the Industry: trusted by all major companies like AWS, Microsoft, Google Cloud and hundreds of others.
Both projects share a lot of similarities, Fluent Bit is fully designed and built on top of the best ideas of Fluentd architecture and general design. Choosing which one to use depends on the end-user needs.
The following table describes a comparison of different areas of the projects:
Scope
Containers / Servers
Embedded Linux / Containers / Servers
Language
C & Ruby
C
Memory
> 60MB
~1MB
Performance
Medium Performance
High Performance
Dependencies
Built as a Ruby Gem, it requires a certain number of gems.
Zero dependencies, unless some special plugin requires them.
Plugins
More than 1000 external plugins are available
More than 100 built-in plugins are available
License
Both Fluentd and Fluent Bit can work as Aggregators or Forwarders, they both can complement each other or use them as standalone solutions. In the recent years, Cloud Providers switched from Fluentd to Fluent Bit for performance and compatibility reasons. Fluent Bit is now considered the next generation solution.
Create flexible routing rules
Routing is a core feature that allows to route your data through Filters and finally to one or multiple destinations. The router relies on the concept of Tags and Matching rules
There are two important concepts in Routing:
Tag
Match
When the data is generated by the input plugins, it comes with a Tag (most of the time the Tag is configured manually), the Tag is a human-readable indicator that helps to identify the data source.
In order to define where the data should be routed, a Match rule must be specified in the output configuration.
Consider the following configuration example that aims to deliver CPU metrics to an Elasticsearch database and Memory metrics to the standard output interface:
Note: the above is a simple example demonstrating how Routing is configured.
Routing works automatically reading the Input Tags and the Output Match rules. If some data has a Tag that doesn't match upon routing time, the data is deleted.
Routing is flexible enough to support wildcard in the Match pattern. The below example defines a common destination for both sources of data:
The match rule is set to my_* which means it will match any Tag that starts with my_.
Routing also provides support for regex with the Match_Regex pattern, allowing for more complex and precise matching criteria. The following example demonstrates how to route data from sources based on a regular expression:
In this configuration, the Match_regex rule is set to .*_sensor_[AB]
. This regular expression will match any Tag that ends with "_sensor_A" or "_sensor_B", regardless of what precedes it. This approach provides a more flexible and powerful way to handle different source tags with a single routing rule.
There are a few key concepts that are really important to understand how Fluent Bit operates.
Before diving into Fluent Bit it’s good to get acquainted with some of the key concepts of the service. This document provides a gentle introduction to those concepts and common Fluent Bit terminology. We’ve provided a list below of all the terms we’ll cover, but we recommend reading this document from start to finish to gain a more general understanding of our log and stream processor.
Event or Record
Filtering
Tag
Timestamp
Match
Structured Message
Every incoming piece of data that belongs to a log or a metric that is retrieved by Fluent Bit is considered an Event or a Record.
As an example consider the following content of a Syslog file:
It contains four lines and all of them represents four independent Events.
Internally an Event is comprised of:
timestamp
key/value metadata (since v2.1.0)
payload
The Fluent Bit wire protocol represents an Event as a 2-element array with a nested array as the first element:
where
TIMESTAMP is a timestamp in seconds as an integer or floating point value (not a string);
METADATA is a possibly-empty object containing event metadata; and
MESSAGE is an object containing the event body.
Fluent Bit versions prior to v2.1.0 instead used:
to represent events. This format is still supported for reading input event streams.
In some cases it is required to perform modifications on the Events content, the process to alter, enrich or drop Events is called Filtering.
There are many use cases when Filtering is required like:
Append specific information to the Event like an IP address or metadata.
Select a specific piece of the Event content.
Drop Events that matches certain pattern.
Every Event that gets into Fluent Bit gets assigned a Tag. This tag is an internal string that is used in a later stage by the Router to decide which Filter or Output phase it must go through.
Most of the tags are assigned manually in the configuration. If a tag is not specified, Fluent Bit will assign the name of the Input plugin instance from where that Event was generated from.
The only input plugin that does NOT assign tags is Forward input. This plugin speaks the Fluentd wire protocol called Forward where every Event already comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the client.
A Tagged record must always have a Matching rule. To learn more about Tags and Matches check the Routing section.
The Timestamp represents the time when an Event was created. Every Event contains a Timestamp associated. The Timestamp is a numeric fractional integer in the format:
It is the number of seconds that have elapsed since the Unix epoch.
Fractional second or one thousand-millionth of a second.
A timestamp always exists, either set by the Input plugin or discovered through a data parsing process.
Fluent Bit allows to deliver your collected and processed Events to one or multiple destinations, this is done through a routing phase. A Match represent a simple rule to select Events where it Tags matches a defined rule.
To learn more about Tags and Matches check the Routing section.
Source events can have or not have a structure. A structure defines a set of keys and values inside the Event message. As an example consider the following two messages:
At a low level both are just an array of bytes, but the Structured message defines keys and values, having a structure helps to implement faster operations on data modifications.
Fluent Bit always handles every Event message as a structured message. For performance reasons, we use a binary serialization data format called MessagePack.
Consider MessagePack as a binary version of JSON on steroids.
The following serves as a guide on how to install/deploy/upgrade Fluent Bit
Kubernetes
Docker
Containers on AWS
CentOS / Red Hat
Ubuntu
Debian
Amazon Linux
Raspbian / Raspberry Pi
Yocto / Embedded Linux
Windows Server 2019
Windows 10 2019.03
macOS
Linux, FreeBSD
macOS
Windows
If you are interested in learning about Fluent Bit you can try out the sandbox environment
Fluent Bit packages are also provided by enterprise providers for older end of life versions, Unix systems, and additional support and features including aspects like CVE backporting. A list provided by fluentbit.io/enterprise is provided below
Modify, Enrich or Drop your records
In production environments we want to have full control of the data we are collecting, filtering is an important feature that allows us to alter the data before delivering it to some destination.
Filtering is implemented through plugins, so each filter available could be used to match, exclude or enrich your logs with some specific metadata.
We support many filters, A common use case for filtering is Kubernetes deployments. Every Pod log needs to get the proper metadata associated
Very similar to the input plugins, Filters run in an instance context, which has its own independent configuration. Configuration keys are often called properties.
For more details about the Filters available and their usage, please refer to the Filters section.
Data processing with reliability
Previously defined in the Buffering concept section, the buffer
phase in the pipeline aims to provide a unified and persistent mechanism to store your data, either using the primary in-memory model or using the filesystem based mode.
The buffer
phase already contains the data in an immutable state, meaning, no other filter can be applied.
Note that buffered data is not raw text, it's in Fluent Bit's internal binary representation.
Fluent Bit offers a buffering mechanism in the file system that acts as a backup system to avoid data loss in case of system failures.
Destinations for your data: databases, cloud services and more!
The output interface allows us to define destinations for the data. Common destinations are remote services, local file system or standard interface with others. Outputs are implemented as plugins and there are many available.
When an output plugin is loaded, an internal instance is created. Every instance has its own independent configuration. Configuration keys are often called properties.
Every output plugin has its own documentation section specifying how it can be used and what properties are available.
For more details, please refer to the Output Plugins section.
uses very low CPU and Memory consumption, it's compatible with most of x86, x86_64, arm32v7, arm64v8 based platforms. In order to build it you need the following components in your system for the build process:
Compiler: GCC or clang
CMake
Flex & Bison: only if you enable the Stream Processor or Record Accessor feature (both enabled by default)
Libyaml development headers and libraries
In the core there are not other dependencies, For certain features that depends on third party components like output plugins with special backend libraries (e.g: kafka), those are included in the main source code repository.
Fluent Bit is supported on Linux on IBM Z(s390x), but the WASM and LUA filter plugins are not.
For production systems, we strongly suggest that you always get the latest stable release of the source code in either zip or tarball format from Github using the following link pattern:
https://github.com/fluent/fluent-bit/archive/refs/tags/v<release version>.tar.gz https://github.com/fluent/fluent-bit/archive/refs/tags/v<release version>.zip
For example for version 1.8.12 the link is the following:
For anyone who aims to contribute to the project by testing or extending the code base, you can get the development version from our GIT repository:
Note that our master branch is where the development of Fluent Bit happens. Since it's a development version, expect issues when compiling or at run time.
We encourage everybody to help us testing every development version, at the end this is what will become stable.
The following operating systems and architectures are supported in Fluent Bit.
From an architecture support perspective, Fluent Bit is fully functional on x86_64, Arm64v8 and Arm32v7 based processors.
Fluent Bit is supported on Linux on IBM Z (s390x) environment with some restrictions but only container images are provided for these targets officially.
Convert Unstructured to Structured messages
Dealing with raw strings or unstructured messages is a constant pain; having a structure is highly desired. Ideally we want to set a structure to the incoming data by the Input Plugins as soon as they are collected:
The Parser allows you to convert from unstructured to structured data. As a demonstrative example consider the following Apache (HTTP Server) log entry:
The above log line is a raw string without format, ideally we would like to give it a structure that can be processed later easily. If the proper configuration is used, the log entry could be converted to:
The following article cover the relevant notes for users upgrading from previous Fluent Bit versions. We aim to cover compatibility changes that you must be aware of.
For more details about changes on each release please refer to the .
Note: release notes will be prepared in advance of a Git tag for a release so an official release should provide both a tag and a release note together to allow users to verify and understand the release contents.
The tag drives the overall binary release process so release binaries (containers/packages) will appear after a tag and its associated release note. This allows users to expect the new release binary to appear and allow/deny/update it as appropriate in their infrastructure.
The td-agent-bit
package is no longer provided after this release. Users should switch to the fluent-bit
package.
If you are migrating from previous version of Fluent Bit please review the following important changes:
Now by default the plugin follows a file from the end once the service starts (old behavior was always read from the beginning). For every file found at start, its followed from it last position, for new files discovered at runtime or rotated, they are read from the beginning.
If you desire to keep the old behavior you can set the option read_from_head
to true.
The project_id of in sent to Google Cloud Logging would be set to the project ID rather than the project number. To learn the difference between Project ID and project number, see for more details.
If you have any existing queries based on the resource's project_id, please update your query accordingly.
The migration from v1.4 to v1.5 is pretty straightforward.
If you are migrating from Fluent Bit v1.3, there are no breaking changes. Just new exciting features to enjoy :)
If you are migrating from Fluent Bit v1.2 to v1.3, there are no breaking changes. If you are upgrading from an older version please review the incremental changes below.
On Fluent Bit v1.2 we have fixed many issues associated with JSON encoding and decoding, for hence when parsing Docker logs is no longer necessary to use decoders. The new Docker parser looks like this:
Note: again, do not use decoders.
We have done improvements also on how Kubernetes Filter handle the stringified log message. If the option Merge_Log is enabled, it will try to handle the log content as a JSON map, if so, it will add the keys to the root map.
In addition, we have fixed and improved the option called Merge_Log_Key. If a merge log succeed, all new keys will be packaged under the key specified by this option, a suggested configuration is as follows:
As an example, if the original log content is the following map:
the final record will be composed as follows:
If you are upgrading from Fluent Bit <= 1.0.x you should take in consideration the following relevant changes when switching to Fluent Bit v1.1 series:
We introduced a new configuration property called Kube_Tag_Prefix to help Tag prefix resolution and address an unexpected behavior that landed in previous versions.
During 1.0.x release cycle, a commit in Tail input plugin changed the default behavior on how the Tag was composed when using the wildcard for expansion generating breaking compatibility with other services. Consider the following configuration example:
The expected behavior is that Tag will be expanded to:
but the change introduced in 1.0 series switched from absolute path to the base file name only:
On Fluent Bit v1.1 release we restored to our default behavior and now the Tag is composed using the absolute path of the monitored file.
Having absolute path in the Tag is relevant for routing and flexible configuration where it also helps to keep compatibility with Fluentd behavior.
This behavior switch in Tail input plugin affects how Filter Kubernetes operates. As you know when the filter is used it needs to perform local metadata lookup that comes from the file names when using Tail as a source. Now with the new Kube_Tag_Prefix option you can specify what's the prefix used in Tail input plugin, for the configuration example above the new configuration will look as follows:
So the proper for Kube_Tag_Prefix value must be composed by Tag prefix set in Tail input plugin plus the converted monitored directory replacing slashes with dots.
uses as its build system. The suggested procedure to prepare the build system consists of the following steps:
CMake >= 3.12
Flex
Bison >= 3
YAML headers
OpenSSL headers
In the following steps you can find exact commands to build and install the project with the default options. If you already know how CMake works you can skip this part and look at the build options available. Note that Fluent Bit requires CMake 3.x. You may need to use
cmake3
instead ofcmake
to complete the following steps on your system.
Change to the build/ directory inside the Fluent Bit sources:
Let configure the project specifying where the root path is located:
Now you are ready to start the compilation process through the simple make command:
to continue installing the binary on the system just do:
it's likely you may need root privileges so you can try to prefixing the command with sudo.
Fluent Bit provides certain options to CMake that can be enabled or disabled when configuring, please refer to the following tables under the General Options, Development Options, Input Plugins and _Output Plugins sections.
The input plugins provides certain features to gather information from a specific source type which can be a network interface, some built-in metric or through a specific input device, the following input plugins are available:
The filter plugins allows to modify, enrich or drop records. The following table describes the filters available on this version:
The output plugins gives the capacity to flush the information to some external interface, service or terminal, the following table describes the output plugins available as of this version:
, ,
, , ,
, ,
,
,
,
,
Fluent Bit can work also on OSX and *BSD systems, but not all plugins will be available on all platforms. Official support will be expanding based on community demand. Fluent Bit may run on older operating systems though will need to be built from source, or use custom packages from .
Parsers are fully configurable and are independently and optionally handled by each input plugin, for more details please refer to the section.
If you enabled keepalive
mode in your configuration, note that this configuration property has been renamed to net.keepalive
. Now all Network I/O keepalive is enabled by default, to learn more about this and other associated configuration properties read the section.
If you use the Elasticsearch output plugin, note the default value of type
. Many versions of Elasticsearch will tolerate this, but ES v5.6 through v6.1 require a type without a leading underscore. See the for more.
Linux
x86_64, Arm64v8
x86_64, Arm64v8
x86_64, Arm64v8
x86_64, Arm64v8
x86_64, Arm64v8
x86_64, Arm64v8
x86_64, Arm64v8
x86_64, Arm64v8
x86_64, Arm64v8
x86_64, Arm64v8
x86_64, Arm64v8
x86_64, Arm64v8
x86_64, Arm64v8
x86_64
Arm32v7
Arm32v7
macOS
*
x86_64, Apple M1
Windows
x86_64, x86
x86_64, x86
FLB_ALL
Enable all features available
No
FLB_JEMALLOC
Use Jemalloc as default memory allocator
No
FLB_TLS
Build with SSL/TLS support
Yes
FLB_BINARY
Build executable
Yes
FLB_EXAMPLES
Build examples
Yes
FLB_SHARED_LIB
Build shared library
Yes
FLB_MTRACE
Enable mtrace support
No
FLB_INOTIFY
Enable Inotify support
Yes
FLB_POSIX_TLS
Force POSIX thread storage
No
FLB_SQLDB
Enable SQL embedded database support
No
FLB_HTTP_SERVER
Enable HTTP Server
No
FLB_LUAJIT
Enable Lua scripting support
Yes
FLB_RECORD_ACCESSOR
Enable record accessor
Yes
FLB_SIGNV4
Enable AWS Signv4 support
Yes
FLB_STATIC_CONF
Build binary using static configuration files. The value of this option must be a directory containing configuration files.
FLB_STREAM_PROCESSOR
Enable Stream Processor
Yes
FLB_CONFIG_YAML
Enable YAML configuration support
Yes
FLB_WASM
Build with WASM runtime support
Yes
FLB_WAMRC
Build with WASM AOT compiler executable
No
FLB_DEBUG
Build binaries with debug symbols
No
FLB_VALGRIND
Enable Valgrind support
No
FLB_TRACE
Enable trace mode
No
FLB_SMALL
Minimise binary size
No
FLB_TESTS_RUNTIME
Enable runtime tests
No
FLB_TESTS_INTERNAL
Enable internal tests
No
FLB_TESTS
Enable tests
No
FLB_BACKTRACE
Enable backtrace/stacktrace support
Yes
FLB_MSGPACK_TO_JSON_INIT_BUFFER_SIZE
Determine initial buffer size for msgpack to json conversion in terms of memory used by payload.
2.0
FLB_MSGPACK_TO_JSON_REALLOC_BUFFER_SIZE
Determine percentage of reallocation size when msgpack to json conversion buffer runs out of memory.
0.1
Enable Collectd input plugin
On
Enable CPU input plugin
On
Enable Disk I/O Metrics input plugin
On
Enable Docker metrics input plugin
On
Enable Exec input plugin
On
Enable Exec WASI input plugin
On
Enable Fluent Bit metrics input plugin
On
Enable Elasticsearch/OpenSearch Bulk input plugin
On
Enable Forward input plugin
On
Enable Head input plugin
On
Enable Health input plugin
On
Enable Kernel log input plugin
On
Enable Memory input plugin
On
Enable MQTT Server input plugin
On
Enable Network I/O metrics input plugin
On
Enable Process monitoring input plugin
On
Enable Random input plugin
On
Enable Serial input plugin
On
Enable Standard input plugin
On
Enable Syslog input plugin
On
Enable Systemd / Journald input plugin
On
Enable Tail (follow files) input plugin
On
Enable TCP input plugin
On
Enable system temperature(s) input plugin
On
Enable UDP input plugin
On
Enable Windows Event Log input plugin (Windows Only)
On
Enable Windows Event Log input plugin using winevt.h API (Windows Only)
On
Enable AWS metadata filter
On
Enable AWS metadata filter
On
FLB_FILTER_EXPECT
Enable Expect data test filter
On
Enable Grep filter
On
Enable Kubernetes metadata filter
On
Enable Lua scripting filter
On
Enable Modify filter
On
Enable Nest filter
On
Enable Parser filter
On
Enable Record Modifier filter
On
Enable Rewrite Tag filter
On
Enable Stdout filter
On
Enable Sysinfo filter
On
Enable Throttle filter
On
Enable Type Converter filter
On
Enable WASM filter
On
Enable Microsoft Azure output plugin
On
Enable Azure Kusto output plugin
On
Enable Google BigQuery output plugin
On
Enable Counter output plugin
On
Enable Amazon CloudWatch output plugin
On
Enable Datadog output plugin
On
Enable Elastic Search output plugin
On
Enable File output plugin
On
Enable Amazon Kinesis Data Firehose output plugin
On
Enable Amazon Kinesis Data Streams output plugin
On
Enable Flowcounter output plugin
On
Enable Fluentd output plugin
On
Enable Gelf output plugin
On
Enable HTTP output plugin
On
Enable InfluxDB output plugin
On
Enable Kafka output
Off
Enable Kafka REST Proxy output plugin
On
FLB_OUT_LIB
Enable Lib output plugin
On
Enable NATS output plugin
On
FLB_OUT_NULL
Enable NULL output plugin
On
FLB_OUT_PGSQL
Enable PostgreSQL output plugin
On
FLB_OUT_PLOT
Enable Plot output plugin
On
FLB_OUT_SLACK
Enable Slack output plugin
On
Enable Amazon S3 output plugin
On
Enable Splunk output plugin
On
Enable Google Stackdriver output plugin
On
Enable STDOUT output plugin
On
FLB_OUT_TCP
Enable TCP/TLS output plugin
On
Enable Treasure Data output plugin
On
Fluent Bit is distributed as fluent-bit package and is available for the latest stable CentOS system.
The following architectures are supported
x86_64
aarch64 / arm64v8
For CentOS 9+ we use CentOS Stream as the canonical base system.
A simple installation script is provided to be used for most Linux targets. This will always install the most recent version released.
This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to follow the instructions below.
CentOS 8 is now EOL so the default Yum repositories are unavailable.
Make sure to configure to use an appropriate mirror, for example:
An alternative is to use Rocky or Alma Linux which should be equivalent.
We provide fluent-bit through a Yum repository. In order to add the repository reference to your system, please add a new file called fluent-bit.repo in /etc/yum.repos.d/ with the following content:
It is best practice to always enable the gpgcheck and repo_gpgcheck for security reasons. We sign our repository metadata as well as all of our packages.
From the 1.9.0 and 1.8.15 releases please note that the GPG key has been updated at https://packages.fluentbit.io/fluentbit.key so ensure this new one is added.
The GPG Key fingerprint of the new key is:
The previous key is still available at https://packages.fluentbit.io/fluentbit-legacy.key and may be required to install previous versions.
The GPG Key fingerprint of the old key is:
Refer to the supported platform documentation to see which platforms are supported in each release.
Once your repository is configured, run the following command to install it:
Now the following step is to instruct Systemd to enable the service:
If you do a status check, you should see a similar output like this:
The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/messages file.
The fluent-bit.repo file for the latest installations of Fluent-Bit uses a $releasever variable to determine the correct version of the package to install to your system:
Depending on your Red Hat distribution version, this variable may return a value other than the OS major release version (e.g., RHEL7 Server distributions return "7Server" instead of just "7"). The Fluent-Bit package url uses just the major OS release version, so any other value here will cause a 404.
In order to resolve this issue, you can replace the $releasever variable with your system's OS major release version. For example:
Fluent Bit is distributed as fluent-bit package and is available for the latest stable Ubuntu system: Jammy Jellyfish.
A simple installation script is provided to be used for most Linux targets. This will always install the most recent version released.
This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to follow the instructions below.
The first step is to add our server GPG key to your keyring to ensure you can get our signed packages. Follow the official Debian wiki guidance: https://wiki.debian.org/DebianRepository/UseThirdParty#OpenPGP\_Key\_distribution
From the 1.9.0 and 1.8.15 releases please note that the GPG key has been updated at https://packages.fluentbit.io/fluentbit.key so ensure this new one is added.
The GPG Key fingerprint of the new key is:
The previous key is still available at https://packages.fluentbit.io/fluentbit-legacy.key and may be required to install previous versions.
The GPG Key fingerprint of the old key is:
Refer to the supported platform documentation to see which platforms are supported in each release.
On Ubuntu, you need to add our APT server entry to your sources lists, please add the following content at bottom of your /etc/apt/sources.list file - ensure to set CODENAME
to your specific Ubuntu release name (e.g. focal
for Ubuntu 20.04):
Now let your system update the apt database:
We recommend upgrading your system (sudo apt-get upgrade
). This could avoid potential issues with expired certificates.
If you have the following error "Certificate verification failed", you might want to check if the package ca-certificates
is properly installed (sudo apt-get install ca-certificates
).
Using the following apt-get command you are able now to install the latest fluent-bit:
Now the following step is to instruct systemd to enable the service:
If you do a status check, you should see a similar output like this:
The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/syslog file.
Fluent Bit is distributed as fluent-bit package and is available for the latest Amazon Linux 2 and Amazon Linux 2023. The following architectures are supported
x86_64
aarch64 / arm64v8
A simple installation script is provided to be used for most Linux targets. This will always install the most recent version released.
This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to follow the instructions below.
Amazon Linux 2022 was previously supported but is removed since it became GA Amazon Linux 2023
We provide fluent-bit through a Yum repository. In order to add the repository reference to your system, please add a new file called fluent-bit.repo in /etc/yum.repos.d/ with the following content:
Note: we encourage you always enable the gpgcheck for security reasons. All our packages are signed.
From the 1.9.0 and 1.8.15 releases please note that the GPG key has been updated at https://packages.fluentbit.io/fluentbit.key so ensure this new one is added.
The GPG Key fingerprint of the new key is:
The previous key is still available at https://packages.fluentbit.io/fluentbit-legacy.key and may be required to install previous versions.
The GPG Key fingerprint of the old key is:
Refer to the supported platform documentation to see which platforms are supported in each release.
Once your repository is configured, run the following command to install it:
Now the following step is to instruct systemd to enable the service:
If you do a status check, you should see a similar output like this:
The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/messages file.
The most secure option is to create the repositories acccording to the instructions for your specific OS.
A simple installation script is provided to be used for most Linux targets. This will by default install the most recent version released.
This is purely a convenience helper and should always be validated prior to use.
From the 1.9.0 and 1.8.15 releases please note that the GPG key has been updated at https://packages.fluentbit.io/fluentbit.key so ensure this new one is added.
The GPG Key fingerprint of the new key is:
The previous key is still available at https://packages.fluentbit.io/fluentbit-legacy.key and may be required to install previous versions.
The GPG Key fingerprint of the old key is:
Refer to the supported platform documentation to see which platforms are supported in each release.
From version 1.9, td-agent-bit
is a deprecated package and is removed after 1.9.9. The correct package name to use now is fluent-bit
.
Fluent Bit container images are available on Docker Hub ready for production usage. Current available images can be deployed in multiple architectures.
Get started by simply typing the following command:
The following table describes the Linux container tags that are available on Docker Hub repository:
It is strongly suggested that you always use the latest image of Fluent Bit.
Windows container images are provided from v2.0.6 for Windows Server 2019 and Windows Server 2022. These can be found as tags on the same Docker Hub registry above.
From a deployment perspective, there is no need to specify an architecture, the container client tool that pulls the image gets the proper layer for the running architecture.
Note: replace cosign
above with the binary installed if it has a different name (e.g. cosign-linux-amd64
).
Keyless signing is also provided but this is still experimental:
Download the last stable image from 2.0 series:
Once the image is in place, now run the following (useless) test which makes Fluent Bit measure CPU usage by the container:
That command will let Fluent Bit measure CPU usage every second and flush the results to the standard output, e.g:
Alpine Linux uses Musl C library instead of Glibc. Musl is not fully compatible with Glibc which generated many issues in the following areas when used with Fluent Bit:
Memory Allocator: to run Fluent Bit properly in high-load environments, we use Jemalloc as a default memory allocator which reduce fragmentation and provides better performance for our needs. Jemalloc cannot run smoothly with Musl and requires extra work.
Alpine Linux Musl functions bootstrap have a compatibility issue when loading Golang shared libraries, this generate problems when trying to load Golang output plugins in Fluent Bit.
Alpine Linux Musl Time format parser does not support Glibc extensions
Maintainers preference in terms of base image due to security and maintenance reasons are Distroless and Debian.
Only include what you need, reduce the attack surface available.
Reduces size so improves perfomance as well.
Reduces false positives on scans (and reduces resources required for scanning).
Reduces supply chain security requirements to just what you need.
Helps prevent unauthorised processes or users interacting with the container.
Less need to harden the container (and container runtime, K8S, etc.).
Faster CICD processes.
With any choice of course there are downsides:
No shell or package manager to update/add things.
Generally though dynamic updating is a bad idea in containers as the time it is done affects the outcome: two containers started at different times using the same base image may perform differently or get different dependencies, etc.
A better approach is to rebuild a new image version but then you can do this with Distroless, however it is harder requiring multistage builds or similar to provide the new dependencies.
Debugging can be harder.
More specifically you need applications set up to properly expose information for debugging rather than rely on traditional debug approaches of connecting to processes or dumping memory. This can be an upfront cost vs a runtime cost but does shift left in the development process so hopefully is a reduction overall.
Assumption that Distroless is secure: nothing is secure (just more or less secure) and there are still exploits so it does not remove the need for securing your system.
Sometimes you need to use a common base image, e.g. with audit/security/health/etc. hooks integrated, or common base tooling (this could still be Distroless though).
One other important thing to note is that exec
'ing into a container will potentially impact resource limits.
This can be a quite different container from the one you want to investigate (e.g. lots of extra tools or even a different base).
No resource limits applied to this container - can be good or bad.
Runs in pod namespaces, just another container that can access everything the others can.
May need architecture of the pod to share volumes, etc.
Requires more recent versions of K8S and the container runtime plus RBAC allowing it.
Learn how to .
Kubernetes Production Grade Log Processor
is a lightweight and extensible Log Processor that comes with full support for Kubernetes:
Process Kubernetes containers logs from the file system or Systemd/Journald.
Enrich logs with Kubernetes Metadata.
Centralize your logs in third party storage services like Elasticsearch, InfluxDB, HTTP, etc.
Before getting started it is important to understand how Fluent Bit will be deployed. Kubernetes manages a cluster of nodes, so our log agent tool will need to run on every node to collect logs from every POD, hence Fluent Bit is deployed as a DaemonSet (a POD that runs on every node of the cluster).
When Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich each entry with the following information (metadata):
Pod Name
Pod ID
Container Name
Container ID
Labels
Annotations
To obtain this information, a built-in filter plugin called kubernetes talks to the Kubernetes API Server to retrieve relevant information such as the pod_id, labels and annotations, other fields such as pod_name, container_id and container_name are retrieved locally from the log file names. All of this is handled automatically, no intervention is required from a configuration aspect.
The recommended way to deploy Fluent Bit is with the official Helm Chart: https://github.com/fluent/helm-charts
If you are using Red Hat OpenShift you will also need to set up security context constraints (SCC):
To add the Fluent Helm Charts repo use the following command
To validate that the repo was added you can run helm search repo fluent
to ensure the charts were added. The default chart can then be installed by running the following
The default configuration of Fluent Bit makes sure of the following:
Consume all containers logs from the running Node.
The Kubernetes filter will enrich the logs with Kubernetes metadata, specifically labels and annotations. The filter only goes to the API Server when it cannot find the cached info, otherwise it uses the cache.
There is an option called Retry_Limit set to False, that means if Fluent Bit cannot flush the records to Elasticsearch it will re-try indefinitely until it succeed.
Fluent Bit by default assumes that logs are formatted by the Docker interface standard. However, when using CRI you can run into issues with malformed JSON if you do not modify the parser used. Fluent Bit includes a CRI log parser that can be used instead. An example of the parser is seen below:
To use this parser change the Input section for your configuration from docker
to cri
Since v1.5.0, Fluent Bit supports deployment to Windows pods.
When deploying Fluent Bit to Kubernetes, there are three log files that you need to pay attention to.
C:\k\kubelet.err.log
This is the error log file from kubelet daemon running on host.
You will need to retain this file for future troubleshooting (to debug deployment failures etc.)
C:\var\log\containers\<pod>_<namespace>_<container>-<docker>.log
This is the main log file you need to watch. Configure Fluent Bit to follow this file.
It is actually a symlink to the Docker log file in C:\ProgramData\
, with some additional metadata on its file name.
C:\ProgramData\Docker\containers\<docker>\<docker>.log
This is the log file produced by Docker.
Normally you don't directly read from this file, but you need to make sure that this file is visible from Fluent Bit.
Typically, your deployment yaml contains the following volume configuration.
DNS_Retries
- Retries N times until the network start working (6)
DNS_Wait_Time
- Lookup interval between network status checks (30)
By default, Fluent Bit waits for 3 minutes (30 seconds x 6 times). If it's not enough for you, tweak the configuration as follows.
Fluent Bit is distributed as fluent-bit package for Windows and as a . Fluent Bit has two flavours of Windows installers: a ZIP archive (for quick testing) and an EXE installer (for system installation).
Not all plugins are supported on Windows: the shows the default set of supported plugins.
Make sure to provide a valid Windows configuration with the installation, a sample one is shown below:
From version 1.9, td-agent-bit
is a deprecated package and was removed after 1.9.9. The correct package name to use now is fluent-bit
.
The latest stable version is 2.2.2. Each version is available via the following download URLs.
Note these are now using the Github Actions built versions, the legacy AppVeyor builds are still available (AMD 32/64 only) at releases.fluentbit.io but are deprecated.
MSI installers are also available:
To check the integrity, use Get-FileHash
cmdlet on PowerShell.
Download a ZIP archive from above. There are installers for 32-bit and 64-bit environments, so choose one suitable for your environment.
Then you need to expand the ZIP archive. You can do this by clicking "Extract All" on Explorer, or if you're using PowerShell, you can use Expand-Archive
cmdlet.
The ZIP package contains the following set of files.
Now, launch cmd.exe or PowerShell on your machine, and execute fluent-bit.exe
as follows.
If you see the following output, it's working fine!
To halt the process, press CTRL-C in the terminal.
Double-click the EXE installer you've downloaded. The installation wizard will automatically start.
Click Next and proceed. By default, Fluent Bit is installed into C:\Program Files\fluent-bit\
, so you should be able to launch fluent-bit as follows after installation.
To silently install to C:\fluent-bit
directory here is an example:
The uninstaller automatically provided also supports a silent un-install using the same /S
flag. This may be useful for provisioning with automation like Ansible, Puppet, etc.
Windows services are equivalent to "daemons" in UNIX (i.e. long-running background processes). Since v1.5.0, Fluent Bit has the native support for Windows Service.
Suppose you have the following installation layout:
To register Fluent Bit as a Windows service, you need to execute the following command on Command Prompt. Please be careful that a single space is required after binpath=
.
Now Fluent Bit can be started and managed as a normal Windows service.
To halt the Fluent Bit service, just execute the "stop" command.
To start Fluent Bit automatically on boot, execute the following:
C:\Program Files
Quotations are required if file paths contain spaces. Here is an example:
Instead of sc.exe
, PowerShell can be used to manage Windows services.
Create a Fluent Bit service:
Start the service:
Query the service status:
Stop the service:
Remove the service (requires PowerShell 6.0 or later)
If you need to create a custom executable, you can use the following procedure to compile Fluent Bit by yourself.
First, you need Microsoft Visual C++ to compile Fluent Bit. You can install the minimum toolkit by the following command:
When asked which packages to install, choose "C++ Build Tools" (make sure that "C++ CMake tools for Windows" is selected too) and wait until the process finishes.
It is important to have installed OpenSSL binaries, at least the library files and headers.
Open the start menu on Windows and type "Command Prompt for VS". From the result list select the one that corresponds to your target system ( x86 or x64).
Note: Check that the installed OpenSSL library files match the selected target. You can check the library files by using the dumpbin command with the /headers option .
Clone the source code of Fluent Bit.
Compile the source code.
Now you should be able to run Fluent Bit:
To create a ZIP package, call cpack
as follows:
Our production stable images are based on focusing on security containing just the Fluent Bit binary and minimal system libraries and basic configuration. We also provide debug images for all architectures (from 1.9.0+) which contain a full (Debian) shell and package manager that can be used to troubleshoot or for testing purposes.
1.9 and 2.0 container images are signed using Cosign/Sigstore. These signatures can be verified using cosign
():
Note: COSIGN_EXPERIMENTAL=1
is used to allow verification of images signed in KEYLESS mode. To learn more about keyless signing, please refer to .
Briefly tackled in a which links out to the following possibly opposing views:
The reasons for using Distroless are fairly well covered here:
For debugging, debug containers are available now in K8S:
Our Kubernetes Filter plugin is fully inspired by the written by .
should be deployed as a DaemonSet, so on that way it will be available on every node of your Kubernetes cluster.
is a package manager for Kubernetes and allows you to quickly deploy application packages into your running cluster. Fluent Bit is distributed via a helm chart found in the Fluent Helm Charts repo: .
The default chart values include configuration to read container logs, with Docker parsing, systemd logs apply Kubernetes metadata enrichment and finally output to an Elasticsearch cluster. You can modify the values file included to specify additional outputs, health checks, monitoring endpoints, or other configuration options.
The will not append more than 5MB into the engine until they are flushed to the Elasticsearch backend. This limit aims to provide a workaround for scenarios.
The default backend in the configuration is Elasticsearch set by the . It uses the Logstash format to ingest the logs. If you need a different Index and Type, please refer to the plugin option and do your own adjustments.
Assuming the basic volume configuration described above, you can apply the following config to start logging. You can visualize this configuration
Windows pods often lack working DNS immediately after boot (). To mitigate this issue, filter_kubernetes
provides a built-in mechanism to wait until the network starts up:
Download an EXE installer from the . It has both 32-bit and 64-bit builds. Choose one which is suitable for you.
The Windows installer is built by [CPack
using NSIS() and so supports the that all NSIS installers do for silent installation and the directory to install to.
Also you need to install flex and bison. One way to install them on Windows is to use .
Add the path C:\WinFlexBison
to your systems environment variable "Path". .
Also you need to install to pull the source code from the repository.
2.2.2-debug
x86_64, arm64v8, arm32v7, s390x
Debug images
2.2.2
x86_64, arm64v8, arm32v7, s390x
Release v2.2.2
2.2.1-debug
x86_64, arm64v8, arm32v7, s390x
Debug images
2.2.1
x86_64, arm64v8, arm32v7, s390x
Release v2.2.1
2.2.0-debug
x86_64, arm64v8, arm32v7
Debug images
2.2.0
x86_64, arm64v8, arm32v7
Release v2.2.0
2.1.10-debug
x86_64, arm64v8, arm32v7
Debug images
2.1.10
x86_64, arm64v8, arm32v7
Release v2.1.10
2.1.9-debug
x86_64, arm64v8, arm32v7
Debug images
2.1.9
x86_64, arm64v8, arm32v7
Release v2.1.9
2.1.8-debug
x86_64, arm64v8, arm32v7
Debug images
2.1.8
x86_64, arm64v8, arm32v7
Release v2.1.8
2.1.7-debug
x86_64, arm64v8, arm32v7
Debug images
2.1.7
x86_64, arm64v8, arm32v7
Release v2.1.7
2.1.6-debug
x86_64, arm64v8, arm32v7
Debug images
2.1.6
x86_64, arm64v8, arm32v7
Release v2.1.6
2.1.5
x86_64, arm64v8, arm32v7
Release v2.1.5
2.1.5-debug
x86_64, arm64v8, arm32v7
Debug images
2.1.4
x86_64, arm64v8, arm32v7
Release v2.1.4
2.1.4-debug
x86_64, arm64v8, arm32v7
Debug images
2.1.3
x86_64, arm64v8, arm32v7
Release v2.1.3
2.1.3-debug
x86_64, arm64v8, arm32v7
Debug images
2.1.2
x86_64, arm64v8, arm32v7
Release v2.1.2
2.1.2-debug
x86_64, arm64v8, arm32v7
Debug images
2.1.1
x86_64, arm64v8, arm32v7
Release v2.1.1
2.1.1-debug
x86_64, arm64v8, arm32v7
v2.1.x releases (production + debug)
2.1.0
x86_64, arm64v8, arm32v7
Release v2.1.0
2.1.0-debug
x86_64, arm64v8, arm32v7
v2.1.x releases (production + debug)
2.0.11
x86_64, arm64v8, arm32v7
Release v2.0.11
2.0.11-debug
x86_64, arm64v8, arm32v7
v2.0.x releases (production + debug)
2.0.10
x86_64, arm64v8, arm32v7
Release v2.0.10
2.0.10-debug
x86_64, arm64v8, arm32v7
v2.0.x releases (production + debug)
2.0.9
x86_64, arm64v8, arm32v7
Release v2.0.9
2.0.9-debug
x86_64, arm64v8, arm32v7
v2.0.x releases (production + debug)
2.0.8
x86_64, arm64v8, arm32v7
Release v2.0.8
2.0.8-debug
x86_64, arm64v8, arm32v7
v2.0.x releases (production + debug)
2.0.6
x86_64, arm64v8, arm32v7
Release v2.0.6
2.0.6-debug
x86_64, arm64v8, arm32v7
v2.0.x releases (production + debug)
2.0.5
x86_64, arm64v8, arm32v7
Release v2.0.5
2.0.5-debug
x86_64, arm64v8, arm32v7
v2.0.x releases (production + debug)
2.0.4
x86_64, arm64v8, arm32v7
Release v2.0.4
2.0.4-debug
x86_64, arm64v8, arm32v7
v2.0.x releases (production + debug)
2.0.3
x86_64, arm64v8, arm32v7
Release v2.0.3
2.0.3-debug
x86_64, arm64v8, arm32v7
v2.0.x releases (production + debug)
2.0.2
x86_64, arm64v8, arm32v7
Release v2.0.2
2.0.2-debug
x86_64, arm64v8, arm32v7
v2.0.x releases (production + debug)
2.0.1
x86_64, arm64v8, arm32v7
Release v2.0.1
2.0.1-debug
x86_64, arm64v8, arm32v7
v2.0.x releases (production + debug)
2.0.0
x86_64, arm64v8, arm32v7
Release v2.0.0
2.0.0-debug
x86_64, arm64v8, arm32v7
v2.0.x releases (production + debug)
1.9.9
x86_64, arm64v8, arm32v7
Release v1.9.9
1.9.9-debug
x86_64, arm64v8, arm32v7
v1.9.x releases (production + debug)
1.9.8
x86_64, arm64v8, arm32v7
Release v1.9.8
1.9.8-debug
x86_64, arm64v8, arm32v7
v1.9.x releases (production + debug)
1.9.7
x86_64, arm64v8, arm32v7
Release v1.9.7
1.9.7-debug
x86_64, arm64v8, arm32v7
v1.9.x releases (production + debug)
1.9.6
x86_64, arm64v8, arm32v7
Release v1.9.6
1.9.6-debug
x86_64, arm64v8, arm32v7
v1.9.x releases (production + debug)
1.9.5
x86_64, arm64v8, arm32v7
Release v1.9.5
1.9.5-debug
x86_64, arm64v8, arm32v7
v1.9.x releases (production + debug)
1.9.4
x86_64, arm64v8, arm32v7
Release v1.9.4
1.9.4-debug
x86_64, arm64v8, arm32v7
v1.9.x releases (production + debug)
1.9.3
x86_64, arm64v8, arm32v7
Release v1.9.3
1.9.3-debug
x86_64, arm64v8, arm32v7
v1.9.x releases (production + debug)
1.9.2
x86_64, arm64v8, arm32v7
Release v1.9.2
1.9.2-debug
x86_64, arm64v8, arm32v7
v1.9.x releases (production + debug)
1.9.1
x86_64, arm64v8, arm32v7
Release v1.9.1
1.9.1-debug
x86_64, arm64v8, arm32v7
v1.9.x releases (production + debug)
1.9.0
x86_64, arm64v8, arm32v7
Release v1.9.0
1.9.0-debug
x86_64, arm64v8, arm32v7
v1.9.x releases (production + debug)
Fluent Bit might optionally use a configuration file to define how the service will behave.
Before proceeding we need to understand how the configuration schema works.
The schema is defined by three concepts:
Sections
Entries: Key/Value
Indented Configuration Mode
A simple example of a configuration file is as follows:
A section is defined by a name or title inside brackets. Looking at the example above, a Service section has been set using [SERVICE] definition. Section rules:
All section content must be indented (4 spaces ideally).
Multiple sections can exist on the same file.
A section is expected to have comments and entries, it cannot be empty.
Any commented line under a section, must be indented too.
End-of-line comments are not supported, only full-line comments.
A section may contain Entries, an entry is defined by a line of text that contains a Key and a Value, using the above example, the [SERVICE]
section contains two entries, one is the key Daemon with value off and the other is the key Log_Level with the value debug. Entries rules:
An entry is defined by a key and a value.
A key must be indented.
A key must contain a value which ends in the breakline.
Multiple keys with the same name can exist.
Also commented lines are set prefixing the # character, those lines are not processed but they must be indented too.
Fluent Bit configuration files are based in a strict Indented Mode, that means that each configuration file must follow the same pattern of alignment from left to right when writing text. By default an indentation level of four spaces from left to right is suggested. Example:
As you can see there are two sections with multiple entries and comments, note also that empty lines are allowed and they do not need to be indented.
Currently, Fluent Bit supports two configuration formats:
Yaml. (YAML configuration is production ready since Fluent Bit 2.0.)
Fluent Bit also supports a CLI interface with various flags matching up to the configuration options available.
Fluent Bit supports the usage of environment variables in any value associated to a key when using a configuration file.
The variables are case sensitive and can be used in the following format:
When Fluent Bit starts, the configuration reader will detect any request for ${MY_VARIABLE}
and will try to resolve its value.
When Fluent Bit is running under systemd (using the official packages), environment variables can be set in the following files:
/etc/default/fluent-bit
(Debian based system)
/etc/sysconfig/fluent-bit
(Others)
These files are ignored if they do not exist.
Create the following configuration file (fluent-bit.conf
):
Open a terminal and set the environment variable:
The above command set the 'stdout' value to the variable
MY_OUTPUT
.
Run Fluent Bit with the recently created configuration file:
As you can see the service worked properly as the configuration was valid.
YAML configuration feature was introduced since FLuent Bit version 1.9 as experimental, and it is production ready since Fluent Bit 2.0.
Fluent Bit source code provides Bitbake recipes to configure, build and package the software for a Yocto based image. Note that specific steps of usage of these recipes in your Yocto environment (Poky) is out of the scope of this documentation.
We distribute two main recipes, one for testing/dev purposes and other with the latest stable release.
devel
Build Fluent Bit from GIT master. This recipe aims to be used for development and testing purposes only.
v1.8.11
Build latest stable version of Fluent Bit.
It's strongly recommended to always use the stable release of Fluent Bit recipe and not the one from GIT master for production deployments.
Fluent Bit >= v1.1.x fully supports x86_64, x86, arm32v7 and arm64v8.
It's common that Fluent Bit aims to connect to external services to deliver the logs over the network, this is the case of , and within others. Being able to connect to one node (host) is normal and enough for more of the use cases, but there are other scenarios where balancing across different nodes is required. The Upstream feature provides such capability.
An Upstream defines a set of nodes that will be targeted by an output plugin, by the nature of the implementation an output plugin must support the Upstream feature. The following plugin(s) have Upstream support:
The current balancing mode implemented is round-robin.
To define an Upstream it's required to create an specific configuration file that contains an UPSTREAM and one or multiple NODE sections. The following table describe the properties associated to each section. Note that all of them are mandatory:
A Node might contain additional configuration keys required by the plugin, on that way we provide enough flexibility for the output plugin, a common use case is Forward output where if TLS is enabled, it requires a shared key (more details in the example below).
In addition to the properties defined in the table above, the network operations against a defined node can optionally be done through the use of TLS for further encryption and certificates use.
The TLS options available are described in the section and can be added to the any Node section.
The following example defines an Upstream called forward-balancing which aims to be used by Forward output plugin, it register three Nodes:
node-1: connects to 127.0.0.1:43000
node-2: connects to 127.0.0.1:44000
node-3: connects to 127.0.0.1:45000 using TLS without verification. It also defines a specific configuration option required by Forward output called shared_key.
Note that every Upstream definition must exists on it own configuration file in the file system. Adding multiple Upstreams in the same file or different files is not allowed.
In an ideal world, applications might log their messages within a single line, but in reality applications generate multiple log messages that sometimes belong to the same context. But when is time to process such information it gets really complex. Consider application stack traces which always have multiple log lines.
Starting from Fluent Bit v1.8, we have implemented a unified Multiline core functionality to solve all the user corner cases. In this section, you will learn about the features and configuration options available.
The Multiline parser engine exposes two ways to configure and use the functionality:
Built-in multiline parser
Configurable multiline parser
Without any extra configuration, Fluent Bit exposes certain pre-configured parsers (built-in) to solve specific multiline parser cases, e.g:
Besides the built-in parsers listed above, through the configuration files is possible to define your own Multiline parsers with their own rules.
A multiline parser is defined in a parsers configuration file by using a [MULTILINE_PARSER]
section definition. The Multiline parser must have a unique name and a type plus other configured properties associated with each type.
To understand which Multiline parser type is required for your use case you have to know beforehand what are the conditions in the content that determines the beginning of a multiline message and the continuation of subsequent lines. We provide a regex based configuration that supports states to handle from the most simple to difficult cases.
Before start configuring your parser you need to know the answer to the following questions:
What is the regular expression (regex) that matches the first line of a multiline message ?
What are the regular expressions (regex) that match the continuation lines of a multiline message ?
When matching regex, we have to define states, some states define the start of a multiline message while others are states for the continuation of multiline messages. You can have multiple continuation states definitions to solve complex cases.
The first regex that matches the start of a multiline message is called start_state, then other regexes continuation lines can have different state names.
A rule specifies how to match a multiline pattern and perform the concatenation. A rule is defined by 3 specific components:
state name
regular expression pattern
next state
A rule might be defined as follows (comments added to simplify the definition) :
In the example above, we have defined two rules, each one has its own state name, regex patterns, and the next state name. Every field that composes a rule must be inside double quotes.
The first rule of state name must always be start_state, and the regex pattern must match the first line of a multiline message, also a next state must be set to specify how the possible continuation lines would look like.
The following example provides a full Fluent Bit configuration file for multiline parsing by using the definition explained above.
Example files content:
This is the primary Fluent Bit configuration file. It includes the parsers_multiline.conf
and tails the file test.log
by applying the multiline parser multiline-regex-test
. Then it sends the processing to the standard output.
This second file defines a multiline parser for the example.
An example file with multiline content:
By running Fluent Bit with the given configuration file you will obtain:
The lines that did not match a pattern are not considered as part of the multiline message, while the ones that matched the rules were concatenated properly.
The multiline parser is a very powerful feature, but it has some limitations that you should be aware of:
The multiline parser is not affected by the buffer_max_size
configuration option, allowing the composed log record to grow beyond this size. Hence, the skip_long_lines
option will not be applied to multiline messages.
It is not possible to get the time key from the body of the multiline message. However, it can be extracted and set as a new key by using a filter.
Fluent-bit supports /pat/m
option. It allows .
matches a new line. It is useful to parse multiline log.
The following example is to get date
and message
from concatenated log.
Example files content:
This is the primary Fluent Bit configuration file. It includes the parsers_multiline.conf
and tails the file test.log
by applying the multiline parser multiline-regex-test
. It also parses concatenated log by applying parser named-capture-test
. Then it sends the processing to the standard output.
This second file defines a multiline parser for the example.
An example file with multiline content:
By running Fluent Bit with the given configuration file you will obtain:
This page describes the yaml configuration file used by Fluent Bit
One of the ways to configure Fluent Bit is using a YAML configuration file that works at a global scope.
The YAML configuration file supports the following sections:
Env
Includes
Service
Pipeline
Inputs
Filters
Outputs
The YAML configuration file does not support the following sections yet:
Parsers
YAML configuration is used in the smoke tests for containers, so an always-correct up-to-date example is here: .
The env section allows the definition of configuration variables that will be used later in the configuration file.
Example:
The includes section allows the files to be merged into the YAML configuration to be identified as a list of filenames. If no path is provided, then the file is assumed to be in a folder relative to the file referencing it.
Example:
The service section defines the global properties of the service. The Service keys available as of this version are described in the following table:
The following is an example of a service section:
A pipeline section will define a complete pipeline configuration, including inputs, filters and outputs subsections.
Each of the subsections for inputs, filters and outputs constitutes an array of maps that has the parameters for each. Most properties are either simple strings or numbers so can be define directly, ie:
This pipeline consists of two inputs; a tail plugin and an http server plugin. Each plugin has its own map in the array of inputs consisting of simple properties. To use more advanced properties that consist of multiple values the property itself can be defined using an array, ie: the record and allowlist_key properties for the record_modifier filter:
In the cases where each value in a list requires two values they must be separated by a space, such as in the record property for the record_modifier filter.
An input section defines a source (related to an input plugin). Here we will describe the base configuration for each input section. Note that each input plugin may add it own configuration keys:
The Name is mandatory and it lets Fluent Bit know which input plugin should be loaded. The Tag is mandatory for all plugins except for the input forward plugin (as it provides dynamic tags).
The following is an example of an input section for the cpu plugin.
A filter section defines a filter (related to a filter plugin). Here we will describe the base configuration for each filter section. Note that each filter plugin may add its own configuration keys:
The Name is mandatory and it lets Fluent Bit know which filter plugin should be loaded. The Match or Match_Regex is mandatory for all plugins. If both are specified, Match_Regex takes precedence.
The following is an example of a filter section for the grep plugin:
The outputs section specify a destination that certain records should follow after a Tag match. Currently, Fluent Bit can route up to 256 OUTPUT plugins. The configuration supports the following keys:
The following is an example of an output section:
The following configuration file example demonstrates how to collect CPU metrics and flush the results every five seconds to the standard output:
In recent versions of Fluent-Bit, the input and output plugins can run in separate threads. In Fluent-Bit 2.1.2, we have implemented a new interface called "processor" to extend the processing capabilities in input and output plugins directly without routing the data. This interface allows users to apply data transformations and filtering to incoming data records before they are processed further in the pipeline.
This functionality is only exposed in YAML configuration and not in classic configuration mode due to the restriction of nested levels of configuration.
The following configuration file example demonstrates the use of processors to change the log record in the input plugin section by adding a new key "hostname" with the value "monox", and we use lua to append the tag to the log record. Also in the ouput plugin section we added a new key named "output" with the value "new data". All these without the need of routing the logs further in the pipeline.
To simplify the configuration of regular expressions, you can use the Rubular web site. We have posted an example by using the regex described above plus a log line that matches the pattern:
The following example files can be located at:
For scheduler and retry details, please check there:
docker
Process a log entry generated by a Docker container engine. This parser supports the concatenation of log entries split by Docker.
cri
Process a log entry generated by CRI-O container engine. Same as the docker parser, it supports concatenation of log entries
go
Process log entries generated by a Go based language application and perform concatenation if multiline messages are detected.
python
Process log entries generated by a Python based language application and perform concatenation if multiline messages are detected.
java
Process log entries generated by a Google Cloud Java language application and perform concatenation if multiline messages are detected.
name
Specify a unique name for the Multiline Parser definition. A good practice is to prefix the name with the word multiline_
to avoid confusion with normal parser's definitions.
type
Set the multiline mode, for now, we support the type regex
.
parser
Name of a pre-defined parser that must be applied to the incoming content before applying the regex rule. If no parser is defined, it's assumed that's a raw text and not a structured message.
Note: when a parser is applied to a raw text, then the regex is applied against a specific key of the structured message by using the key_content
configuration property (see below).
key_content
For an incoming structured message, specify the key that contains the data that should be processed by the regular expression and possibly concatenated.
flush_timeout
Timeout in milliseconds to flush a non-terminated multiline buffer. Default is set to 5 seconds.
5s
rule
Configure a rule to match a multiline pattern. The rule has a specific format described below. Multiple rules can be defined.
flush
Set the flush time in seconds.nanoseconds
. The engine loop uses a Flush timeout to define when is required to flush the records ingested by input plugins through the defined output plugins.
5
grace
Set the grace time in seconds
as an Integer value. The engine loop uses a Grace timeout to define the wait time on exit
5
daemon
Boolean value to set if Fluent Bit should run as a Daemon (background) or not. Allowed values are: yes, no, on, and off. note: If you are using a Systemd based unit like the one we provide in our packages, do not turn on this option.
Off
dns.mode
Sets the primary transport layer protocol used by the asynchronous DNS resolver, which can be overridden on a per plugin basis
UDP
log_file
Absolute path for an optional log file. By default, all logs are redirected to the standard error interface (stderr).
log_level
Set the logging verbosity level. Allowed values are: off, error, warn, info, debug and trace. Values are accumulative, e.g., if 'debug' is set, it will include error, warning, info, and debug. Note that trace mode is only available if Fluent Bit was built with the WITH_TRACE option enabled.
info
parsers_file
Path for a parsers
configuration file. Only a single entry is currently supported.
plugins_file
Path for a plugins
configuration file. A plugins configuration file allows the definition of paths for external plugins; for an example, see here.
streams_file
Path for the Stream Processor configuration file. To learn more about Stream Processing configuration go here.
http_server
Enable built-in HTTP Server
Off
http_listen
Set listening interface for HTTP Server when it's enabled
0.0.0.0
http_port
Set TCP Port for the HTTP Server
2020
coro_stack_size
Set the coroutines stack size in bytes. The value must be greater than the page size of the running system. Don't set too small a value (say 4096), or coroutine threads can overrun the stack buffer. Do not change the default value of this parameter unless you know what you are doing.
24576
scheduler.cap
Set a maximum retry time in seconds. The property is supported from v1.8.7.
2000
scheduler.base
Sets the base of exponential backoff. The property is supported from v1.8.7.
5
json.convert_nan_to_null
If enabled, NaN is converted to null when fluent-bit converts msgpack to json.
false
sp.convert_from_str_to_num
If enabled, Stream processor converts from number string to number type.
true
Name
Name of the input plugin. Defined as subsection of the inputs section.
Tag
Tag name associated to all records coming from this plugin.
Log_Level
Set the plugin's logging verbosity level. Allowed values are: off, error, warn, info, debug and trace. Defaults to the SERVICE section's Log_Level.
Name
Name of the filter plugin. Defined as a subsection of the filters section.
Match
A pattern to match against the tags of incoming records. It's case-sensitive and supports the star (*) character as a wildcard.
Match_Regex
A regular expression to match against the tags of incoming records. Use this option if you want to use the full regex syntax.
Log_Level
Set the plugin's logging verbosity level. Allowed values are: off, error, warn, info, debug and trace. Defaults to the SERVICE section's Log_Level.
Name
Name of the output plugin. Defined as a subsection of the outputs section.
Match
A pattern to match against the tags of incoming records. It's case-sensitive and supports the star (*) character as a wildcard.
Match_Regex
A regular expression to match against the tags of incoming records. Use this option if you want to use the full regex syntax.
Log_Level
Set the plugin's logging verbosity level. Allowed values are: off, error, warn, info, debug and trace. The output log level defaults to the SERVICE section's Log_Level.
UPSTREAM
name
Defines a name for the Upstream in question.
NODE
name
Defines a name for the Node in question.
host
IP address or hostname of the target host.
port
TCP port of the target service.
Under certain scenarios it is possible for logs or data to be ingested or created faster than the ability to flush it to some destinations. One such common scenario is when reading from big log files, especially with a large backlog, and dispatching the logs to a backend over the network, which takes time to respond. This generates backpressure leading to high memory consumption in the service.
In order to avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts the amount of data that an input plugin can ingest, this is done through the configuration parameters Mem_Buf_Limit and storage.Max_Chunks_Up.
As described in the Buffering concepts section, Fluent Bit offers two modes for data handling: in-memory only (default) and in-memory + filesystem (optional).
The default storage.type memory
buffer can be restricted with Mem_Buf_Limit. If memory reaches this limit and you reach a backpressure scenario, you will not be able to ingest more data until the data chunks that are in memory can be flushed. The input will be paused and Fluent Bit will emit a [warn] [input] {input name or alias} paused (mem buf overlimit)
log message. Depending on the input plugin in use, this might lead to discard incoming data (e.g: TCP input plugin). The tail plugin can handle pause without data loss; it will store its current file offset and resume reading later. When buffer memory is available, the input will resume collecting/accepting logs and Fluent Bit will emit a [info] [input] {input name or alias} resume (mem buf overlimit)
message.
This risk of data loss can be mitigated by configuring secondary storage on the filesystem using the storage.type
of filesystem
(as described in Buffering & Storage). Initially, logs will be buffered to both memory and filesystem. When the storage.max_chunks_up
limit is reached, all the new data will be stored safely only in the filesystem. Fluent Bit will stop enqueueing new data in memory and will only buffer to the filesystem. Please note that when storage.type filesystem
is set, the Mem_Buf_Limit
setting no longer has any effect, instead, the [SERVICE]
level storage.max_chunks_up
setting controls the size of the memory buffer.
This option is disabled by default and can be applied to all input plugins. Please note that Mem_Buf_Limit
only applies with the default storage.type memory
. Let's explain its behavior using the following scenario:
Mem_Buf_Limit is set to 1MB (one megabyte)
input plugin tries to append 700KB
engine route the data to an output plugin
output plugin backend (HTTP Server) is down
engine scheduler will retry the flush after 10 seconds
input plugin tries to append 500KB
At this exact point, the engine will allow appending those 500KB of data into the memory; in total it will have 1.2MB of data buffered. The limit is permissive and will allow a single write past the limit, but once the limit is exceeded the following actions are taken:
block local buffers for the input plugin (cannot append more data)
notify the input plugin invoking a pause callback
The engine will protect itself and will not append more data coming from the input plugin in question; note that it is the responsibility of the plugin to keep state and decide what to do in that paused state.
After some time, usually measured in seconds, if the scheduler was able to flush the initial 700KB of data or it has given up after retrying, that amount of memory is released and the following actions will occur:
Upon data buffer release (700KB), the internal counters get updated
Counters now are set at 500KB
Since 500KB is < 1MB it checks the input plugin state
If the plugin is paused, it invokes a resume callback
input plugin can continue appending more data
Please note that when storage.type filesystem
is set, the Mem_Buf_Limit
setting no longer has any effect, instead, the [SERVICE]
level storage.max_chunks_up
setting controls the size of the memory buffer.
The setting behaves similarly to the above scenario with Mem_Buf_Limit
when the non-default storage.pause_on_chunks_overlimit
is enabled.
When (default) storage.pause_on_chunks_overlimit
is disabled, the input will not pause when the memory limit is reached. Instead, it will switch to only buffering logs in the filesystem. The disk spaced used for filesystem buffering can be limited with storage.total_limit_size
.
Please consule the Buffering & Storage docs for more information.
Each plugin is independent and not all of them implements the pause and resume callbacks. As said, these callbacks are just a notification mechanism for the plugin.
One example of a plugin that implements these callbacks and keeps state correctly is the Tail Input plugin. When the pause callback is triggered, it pauses its collectors and stops appending data. Upon resume, it resumes the collectors and continues ingesting data. Tail will track the current file offset when it pauses and resume at the same position. If the file has not been deleted or moved, it can still be read.
With the default storage.type memory
and Mem_Buf_Limit
, the following log messages will be emitted for pause and resume:
With storage.type filesystem
and storage.max_chunks_up
, the following log messages will be emitted for pause and resume:
In certain scenarios it would be ideal to estimate how much memory Fluent Bit could be using, this is very useful for containerized environments where memory limits are a must.
In order to that we will assume that the input plugins have set the Mem_Buf_Limit option (you can learn more about it in the Backpressure section).
Input plugins append data independently, so in order to do an estimation, a limit should be imposed through the Mem_Buf_Limit option. If the limit was set to 10MB we need to estimate that in the worse case, the output plugin likely could use 20MB.
Fluent Bit has an internal binary representation for the data being processed, but when this data reaches an output plugin, it will likely create its own representation in a new memory buffer for processing. The best examples are the InfluxDB and Elasticsearch output plugins, both need to convert the binary representation to their respective custom JSON formats before it can be sent to the backend servers.
So, if we impose a limit of 10MB for the input plugins and consider the worse case scenario of the output plugin consuming 20MB extra, as a minimum we need (30MB x 1.2) = 36MB.
It is well known that in intensive environments where memory allocations happen in the orders of magnitude, the default memory allocator provided by Glibc could lead to high fragmentation, reporting a high memory usage by the service.
It's strongly suggested that in any production environment, Fluent Bit should be built with jemalloc enabled (e.g. -DFLB_JEMALLOC=On
). Jemalloc is an alternative memory allocator that can reduce fragmentation (among others things) resulting in better performance.
You can check if Fluent Bit has been built with Jemalloc using the following command:
The output should look like:
If the FLB_HAVE_JEMALLOC option is listed in Build Flags, everything will be fine.
Fluent Bit provides integrated support for Transport Layer Security (TLS) and it predecessor Secure Sockets Layer (SSL) respectively. In this section we will refer as TLS only for both implementations.
Both input and output plugins that perform Network I/O can optionally enable TLS and configure the behavior. The following table describes the properties available:
tls
enable or disable TLS support
Off
tls.verify
force certificate validation
On
tls.debug
Set TLS debug verbosity level. It accept the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational) and 4 Verbose
1
tls.ca_file
absolute path to CA certificate file
tls.ca_path
absolute path to scan for certificate files
tls.crt_file
absolute path to Certificate file
tls.key_file
absolute path to private Key file
tls.key_passwd
optional password for tls.key_file file
tls.vhost
hostname to be used for TLS SNI extension
Note : in order to use TLS on input plugins the user is expected to provide both a certificate and private key
The listed properties can be enabled in the configuration file, specifically on each output plugin section or directly through the command line.
The following output plugins can take advantage of the TLS feature:
The following input plugins can take advantage of the TLS feature:
In addition, other plugins implements a sub-set of TLS support, meaning, with restricted configuration:
By default HTTP input plugin uses plain TCP, enabling TLS from the command line can be done with:
In the command line above, the two properties tls and tls.verify where enabled for demonstration purposes (we strongly suggest always keep verification ON).
The same behavior can be accomplished using a configuration file:
By default HTTP output plugin uses plain TCP, enabling TLS from the command line can be done with:
In the command line above, the two properties tls and tls.verify where enabled for demonstration purposes (we strongly suggest always keep verification ON).
The same behavior can be accomplished using a configuration file:
This will generate a 4096 bit RSA key pair and a certificate that is signed using SHA-256 with the expiration date set to 30 days in the future, test.host.net
set as common name and since we opted out of DES
the private key will be stored in plain text.
Fluent Bit supports TLS server name indication. If you are serving multiple hostnames on a single IP address (a.k.a. virtual hosting), you can make use of tls.vhost
to connect to a specific hostname.
Fluent Bit has an Engine that helps to coordinate the data ingestion from input plugins and calls the Scheduler to decide when it is time to flush the data through one or multiple output plugins. The Scheduler flushes new data at a fixed time of seconds and the Scheduler retries when asked.
Once an output plugin gets called to flush some data, after processing that data it can notify the Engine three possible return statuses:
OK
Retry
Error
If the return status was OK, it means it was successfully able to process and flush the data. If it returned an Error status, it means that an unrecoverable error happened and the engine should not try to flush that data again. If a Retry was requested, the Engine will ask the Scheduler to retry to flush that data, the Scheduler will decide how many seconds to wait before that happens.
The Scheduler provides two configuration options called scheduler.cap and scheduler.base which can be set in the Service section.
scheduler.cap
Set a maximum retry time in seconds. The property is supported from v1.8.7.
2000
scheduler.base
Set a base of exponential backoff. The property is supported from v1.8.7.
5
These two configuration options determine the waiting time before a retry will happen.
Fluent Bit uses an exponential backoff and jitter algorithm to determine the waiting time before a retry.
The waiting time is a random number between a configurable upper and lower bound.
For the Nth retry, the lower bound of the random number will be:
base
The upper bound will be:
min(base * (Nth power of 2), cap)
Given an example where base
is set to 3 and cap
is set to 30.
1st retry: The lower bound will be 3, the upper bound will be 3 * 2 = 6. So the waiting time will be a random number between (3, 6).
2nd retry: the lower bound will be 3, the upper bound will be 3 * (2 * 2) = 12. So the waiting time will be a random number between (3, 12).
3rd retry: the lower bound will be 3, the upper bound will be 3 * (2 * 2 * 2) = 24. So the waiting time will be a random number between (3, 24).
4th retry: the lower bound will be 3, since 3 * (2 * 2 * 2 * 2) = 48 > 30, the upper bound will be 30. So the waiting time will be a random number between (3, 30).
Basically, the scheduler.base determines the lower bound of time between each retry and the scheduler.cap determines the upper bound.
For a detailed explanation of the exponential backoff and jitter algorithm, please check this blog.
The following example configures the scheduler.base as 3 seconds and scheduler.cap as 30 seconds.
The waiting time will be:
1
(3, 6)
2
(3, 12)
3
(3, 24)
4
(3, 30)
The Scheduler provides a simple configuration option called Retry_Limit, which can be set independently on each output section. This option allows us to disable retries or impose a limit to try N times and then discard the data after reaching that limit:
Retry_Limit
N
Integer value to set the maximum number of retries allowed. N must be >= 1 (default: 1)
Retry_Limit
no_limits
or False
When Retry_Limit is set to no_limits
orFalse
, means that there is not limit for the number of retries that the Scheduler can do.
Retry_Limit
no_retries
When Retry_Limit is set to no_retries, means that retries are disabled and Scheduler would not try to send data to the destination if it failed the first time.
The following example configures two outputs where the HTTP plugin has an unlimited number of while the Elasticsearch plugin have a limit of 5 retries:
Fluent Bit implements a unified networking interface that is exposed to components like plugins. This interface abstract all the complexity of general I/O and is fully configurable.
A common use case is when a component or plugin needs to connect to a service to send and receive data. Despite the operational mode sounds easy to deal with, there are many factors that can make things hard like unresponsive services, networking latency or any kind of connectivity error. The networking interface aims to abstract and simplify the network I/O handling, minimize risks and optimize performance.
Most of the time creating a new TCP connection to a remote server is straightforward and takes a few milliseconds. But there are cases where DNS resolving, slow network or incomplete TLS handshakes might create long delays, or incomplete connection statuses.
The net.connect_timeout
allows to configure the maximum time to wait for a connection to be established, note that this value already considers the TLS handshake process.
The net.connect_timeout_log_error
indicates if an error should be logged in case of connect timeout. If disabled, the timeout is logged as debug level message instead.
On environments with multiple network interfaces, might be desired to choose which interface to use for our data that will flow through the network.
The net.source_address
allows to specify which network address must be used for a TCP connection and data flow.
TCP is a connected oriented channel, to deliver and receive data from a remote end-point in most of cases we use a TCP connection. This TCP connection can be created and destroyed once is not longer needed, this approach has pros and cons, here we will refer to the opposite case: keep the connection open.
The concept of Connection Keepalive
refers to the ability of the client (Fluent Bit on this case) to keep the TCP connection open in a persistent way, that means that once the connection is created and used, instead of close it, it can be recycled. This feature offers many benefits in terms of performance since communication channels are always established before hand.
Any component that uses TCP channels like HTTP or TLS, can take advantage of this feature. For configuration purposes use the net.keepalive
property.
If a connection is keepalive enabled, there might be scenarios where the connection can be unused for long periods of time. Having an idle keepalive connection is not helpful and is recommendable to keep them alive if they are used.
In order to control how long a keepalive connection can be idle, we expose the configuration property called net.keepalive_idle_timeout
.
If a transport layer protocol is specified, the plugin whose configuration section the net.dns.mode
setting is specified on overrides the global dns.mode
value and issues DNS requests using the specified protocol which can be either TCP or UDP
By default, Fluent Bit tries to deliver data as faster as possible and create TCP connections on-demand and in keepalive mode for performance reasons. In high-scalable environments, the user might want to control how many connections are done in parallel by setting a limit.
This can be done by the configuration property called net.max_worker_connections
that can be used in the output plugins sections. This feature acts at the worker level, e.g., if you have 5 workers and net.max_worker_connections
is set to 10, a max of 50 connections will be allowed. If the limit is reached, the output plugin will issue a retry.
For plugins that rely on networking I/O, the following section describes the network configuration properties available and how they can be used to optimize performance or adjust to different configuration needs:
net.connect_timeout
Set maximum time expressed in seconds to wait for a TCP connection to be established, this include the TLS handshake time.
10
net.connect_timeout_log_error
On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message.
true
net.dns.mode
Select the primary DNS connection type (TCP or UDP). Can be set in the [SERVICE] section and overridden on a per plugin basis if desired.
net.dns.prefer_ipv4
Prioritize IPv4 DNS results when trying to establish a connection.
false
net.dns.resolver
Select the primary DNS resolver type (LEGACY or ASYNC).
net.keepalive
Enable or disable connection keepalive support. Accepts a boolean value: on / off.
on
net.keepalive_idle_timeout
Set maximum time expressed in seconds for an idle keepalive connection.
30
net.keepalive_max_recycle
Set maximum number of times a keepalive connection can be used before it is retired.
2000
net.max_worker_connections
Set maximum number of TCP connections that can be established per worker.
0 (unlimited)
net.source_address
Specify network address to bind for data traffic.
As an example, we will send 5 random messages through a TCP output connection, in the remote side we will use nc
(netcat) utility to see the data.
Put the following configuration snippet in a file called fluent-bit.conf
:
In another terminal, start nc
and make it listen for messages on TCP port 9090:
Now start Fluent Bit with the configuration file written above and you will see the data flowing to netcat:
If the net.keepalive
option is not enabled, Fluent Bit will close the TCP connection and netcat will quit, here we can see how the keepalive connection works.
After the 5 records arrive, the connection will keep idle and after 10 seconds it will be closed due to net.keepalive_idle_timeout
.
The end-goal of Fluent Bit is to collect, parse, filter and ship logs to a central place. In this workflow there are many phases and one of the critical pieces is the ability to do buffering : a mechanism to place processed data into a temporary location until is ready to be shipped.
By default when Fluent Bit processes data, it uses Memory as a primary and temporary place to store the records, but there are certain scenarios where it would be ideal to have a persistent buffering mechanism based in the filesystem to provide aggregation and data safety capabilities.
Choosing the right configuration is critical and the behavior of the service can be conditioned based in the backpressure settings. Before we jump into the configuration let's make sure we understand the relationship between Chunks, Memory, Filesystem and Backpressure.
Understanding the chunks, buffering and backpressure concepts is critical for a proper configuration. Let's do a recap of the meaning of these concepts.
When an input plugin (source) emits records, the engine groups the records together in a Chunk. A Chunk size usually is around 2MB. By configuration, the engine decides where to place this Chunk, the default is that all chunks are created only in memory.
There are two scenarios where fluent-bit marks chunks as irrecoverable:
When Fluent Bit encounters a bad layout in a chunk. A bad layout is a chunk that does not conform to the expected format. Chunk definition
When Fluent Bit encounters an incorrect or invalid chunk header size.
In both scenarios Fluent-Bit will log an error message and then discard the irrecoverable chunks.
As mentioned above, the Chunks generated by the engine are placed in memory but this is configurable.
If memory is the only mechanism set for the input plugin, it will just store data as much as it can there (memory). This is the fastest mechanism with the least system overhead, but if the service is not able to deliver the records fast enough because of a slow network or an unresponsive remote service, Fluent Bit memory usage will increase since it will accumulate more data than it can deliver.
In a high load environment with backpressure the risks of having high memory usage is the chance of getting killed by the Kernel (OOM Killer). A workaround for this backpressure scenario is to limit the amount of memory in records that an input plugin can register, this configuration property is called mem_buf_limit
. If a plugin has enqueued more than the mem_buf_limit
, it won't be able to ingest more until that data can be delivered or flushed properly. In this scenario the input plugin in question is paused. When the input is paused, records will not be ingested until it is resumed. For some inputs, such as TCP and tail, pausing the input will almost certainly lead to log loss. For the tail input, Fluent Bit can save its current offset in the current file it is reading, and pick back up when the input is resumed.
Look for messages in the Fluent Bit log output like:
The workaround of mem_buf_limit
is good for certain scenarios and environments, it helps to control the memory usage of the service, but at the costs that if a file gets rotated while paused, you might lose that data since it won't be able to register new records. This can happen with any input source plugin. The goal of mem_buf_limit
is memory control and survival of the service.
For full data safety guarantee, use filesystem buffering.
Here is an example input definition:
If this input uses more than 50MB memory to buffer logs, you will get a warning like this in the Fluent Bit logs:
Mem_Buf_Limit
applies only when storage.type
is set to the default value of memory
.
The following section explains the applicable limits when you enable storage.type filesystem
.
Filesystem buffering enabled helps with backpressure and overall memory control.
Behind the scenes, Memory and Filesystem buffering mechanisms are not mutually exclusive. Indeed when enabling filesystem buffering for your input plugin (source) you are getting the best of the two worlds: performance and data safety.
When Filesystem buffering is enabled, the behavior of the engine is different. Upon Chunk creation, the engine stores the content in memory and also maps a copy on disk (through mmap(2)). The newly created Chunk is (1) active in memory, (2) backed up on disk, and (3) is called to be up
which means "the chunk content is up in memory".
How does the Filesystem buffering mechanism deal with high memory usage and backpressure? Fluent Bit controls the number of Chunks that are up
in memory.
By default, the engine allows us to have 128 Chunks up
in memory in total (considering all Chunks), this value is controlled by service property storage.max_chunks_up
. The active Chunks that are up
are ready for delivery and the ones that are still receiving records. Any other remaining Chunk is in a down
state, which means that it is only in the filesystem and won't be up
in memory unless it is ready to be delivered. Remember, chunks are never much larger than 2 MB, thus, with the default storage.max_chunks_up
value of 128, each input is limited to roughly 256 MB of memory.
If the input plugin has enabled storage.type
as filesystem
, when reaching the storage.max_chunks_up
threshold, instead of the plugin being paused, all new data will go to Chunks that are down
in the filesystem. This allows us to control the memory usage by the service and also provides a guarantee that the service won't lose any data. By default, the enforcement of the storage.max_chunks_up
limit is best-effort. Fluent Bit can only append new data to chunks that are up
; when the limit is reached chunks will be temporarily brought up
in memory to ingest new data, and then put to a down
state afterwards. In general, Fluent Bit will work to keep the total number of up
chunks at or below storage.max_chunks_up
.
If storage.pause_on_chunks_overlimit
is enabled (default is off), the input plugin will be paused upon exceeding storage.max_chunks_up
. Thus, with this option, storage.max_chunks_up
becomes a hard limit for the input. When the input is paused, records will not be ingested until it is resumed. For some inputs, such as TCP and tail, pausing the input will almost certainly lead to log loss. For the tail input, Fluent Bit can save its current offset in the current file it is reading, and pick back up when the input is resumed.
Look for messages in the Fluent Bit log output like:
Limiting Filesystem space for Chunks
Fluent Bit implements the concept of logical queues: based on its Tag, a Chunk can be routed to multiple destinations. Thus, we keep an internal reference from where a Chunk was created and where it needs to go.
It's common to find cases where if we have multiple destinations for a Chunk, one of the destinations might be slower than the other, or maybe one is generating backpressure and not all of them. In this scenario, how do we limit the amount of filesystem Chunks that we are logically queueing?
Starting from Fluent Bit v1.6, we introduced the new configuration property for output plugins called storage.total_limit_size
which limits the total size in bytes of chunks that can exist in the filesystem for a certain logical output destination. If one of the destinations reaches the configured storage.total_limit_size
, the oldest Chunk from its queue for that logical output destination will be discarded to make room for new data.
The storage layer configuration takes place in three areas:
Service Section
Input Section
Output Section
The known Service section configures a global environment for the storage layer, the Input sections define which buffering mechanism to use and the output the limits for the logical filesystem queues.
The Service section refers to the section defined in the main configuration file:
storage.path
Set an optional location in the file system to store streams and chunks of data. If this parameter is not set, Input plugins can only use in-memory buffering.
storage.sync
normal
storage.checksum
Enable the data integrity check when writing and reading data from the filesystem. The storage layer uses the CRC32 algorithm.
Off
storage.max_chunks_up
If the input plugin has enabled filesystem
storage type, this property sets the maximum number of Chunks that can be up
in memory. This is the setting to use to control memory usage when you enable storage.type filesystem
.
128
storage.backlog.mem_limit
If storage.path is set, Fluent Bit will look for data chunks that were not delivered and are still in the storage layer, these are called backlog data. Backlog chunks are filesystem chunks that were left over from a previous Fluent Bit run; chunks that could not be sent before exit that Fluent Bit will pick up when restarted. Fluent Bit will check the storage.backlog.mem_limit
value against the current memory usage from all up
chunks for the input. If the up
chunks currently consume less memory than the limit, it will bring the backlog chunks up into memory so they can be sent by outputs.
5M
storage.metrics
off
storage.delete_irrecoverable_chunks
Off
a Service section will look like this:
that configuration sets an optional buffering mechanism where the route to the data is /var/log/flb-storage/, it will use normal synchronization mode, without running a checksum and up to a maximum of 5MB of memory when processing backlog data.
Optionally, any Input plugin can configure their storage preference, the following table describes the options available:
storage.type
Specifies the buffering mechanism to use. It can be memory or filesystem.
memory
storage.pause_on_chunks_overlimit
Specifies if the input plugin should be paused (stop ingesting new data) when the storage.max_chunks_up
value is reached.
off
The following example configures a service that offers filesystem buffering capabilities and two Input plugins being the first based in filesystem and the second with memory only.
If certain chunks are filesystem storage.type based, it's possible to control the size of the logical queue for an output plugin. The following table describes the options available:
storage.total_limit_size
Limit the maximum disk space size in bytes for buffering chunks in the filesystem for the current output logical destination.
The following example create records with CPU usage samples in the filesystem and then they are delivered to Google Stackdriver service limiting the logical queue (buffering) to 5M:
If for some reason Fluent Bit gets offline because of a network issue, it will continue buffering CPU samples but just keep a maximum of 5MB of the newest data.
Every project has a story
On 2014, the team at was forecasting the need for a lightweight log processor for constraint environments like Embedded Linux and Gateways, the project aimed to be part of the Fluentd Ecosystem; at that moment, Eduardo created , a new open source solution written from scratch available under the terms of the .\
After the project was around for some time, it got more traction for normal Linux systems, also with the new containerized world, the Cloud Native community asked to extend the project scope to support more sources, filters, and destinations. Not so long after, Fluent Bit became one of the preferred solutions to solve the logging challenges in Cloud environments.
Fluent Bit is a powerful log processing tool that can deal with different sources and formats, in addition it provides several filters that can be used to perform custom modifications. This flexibility is really good but while your pipeline grows, it's strongly recommended to validate your data and structure.
We encourage Fluent Bit users to integrate data validation in their CI systems
A simplified view of our data processing pipeline is as follows:
In a normal production environment, many Inputs, Filters, and Outputs are defined in the configuration, so integrating a continuous validation of your configuration against expected results is a must. For this requirement, Fluent Bit provides a specific Filter called Expect which can be used to validate expected Keys and Values from your records and takes some action when an exception is found.
Ideally you want to add checkpoints of validation of your data between each step so you can know if your data structure is correct, we do this by using expect filter.
Expect filter sets rules that aims to validate certain criteria like:
does the record contain a key A ?
does the record not contains key A?
does the record key A value equals NULL ?
does the record key A value a different value than NULL ?
does the record key A value equals B ?
Every expect filter configuration can expose specific rules to validate the content of your records, it supports the following configuration properties:
Consider the following JSON file called data.log
with the following content:
The following Fluent Bit configuration file will configure a pipeline to consume the log above apply an expect filter to validate that keys color
and label
exists:
note that if for some reason the JSON parser failed or is missing in the tail
input (line 9), the expect
filter will trigger the exit
action. As a test, go ahead and comment out or remove line 9.
As a second step, we will extend our pipeline and we will add a grep filter to match records that map label
contains a key called name
with value abc
, then an expect filter to re-validate that condition:
When deploying your configuration in production, you might want to remove the expect filters from your configuration since it's an unnecessary extra work unless you want to have a 100% coverage of checks at runtime.
You may wish to test a logging pipeline locally to observe how it deals with log messages. The following is a walk-through for running Fluent Bit and Elasticsearch locally with which can serve as an example for testing other plugins locally.
Refer to the to create a configuration to test.
fluent-bit.conf
:
Use to run Fluent Bit (with the configuration file mounted) and Elasticsearch.
docker-compose.yaml
:
To view indexed logs run:
To "start fresh", delete the index by running:
Enable hot reload through SIGHUP signal or an HTTP endpoint
Fluent Bit supports the hot reloading feature when enabled via the configuration file or command line with -Y
or --enable-hot-reload
option.
To get started with reloading via HTTP, the first step is to enable the HTTP Server from the configuration file:
The above configuration snippet will enable the HTTP endpoint for hot reloading.
Hot reloading can be kicked via HTTP endpoints that are:
PUT /api/v2/reload
POST /api/v2/reload
If users don't enable the hot reloading feature, hot reloading via these endpoints will not work.
For using curl to reload Fluent Bit, users must specify an empty request body as:
Hot reloading also can be kicked via SIGHUP
.
SIGHUP
signal is not supported on Windows. So, users can't enable this feature on Windows.
The number of hot reloaded count can be obtained via the HTTP endpoint that is:
GET /api/v2/reload
The endpoint returns the count of hot-reloaded as follows:
The default value of that number is 0.
The hot reloading feature is currently working on Linux, macOS and Windows.
Configure the synchronization mode used to store the data into the file system. It can take the values normal or full. Using full increases the reliability of the filesystem buffer and ensures that data is guaranteed to be synced to the filesystem even if Fluent Bit crashes. On linux, full corresponds with the MAP_SYNC
option for .
If http_server
option has been enabled in the main [SERVICE]
section, this option registers a new endpoint where internal metrics of the storage layer can be consumed. For more details refer to the section.
When enabled, will be deleted during runtime, and any other irrecoverable chunk located in the configured storage path directory will be deleted when Fluent-Bit starts.
As an example, consider the following pipeline where your source of data is a normal file with JSON content on it and then two filters: to exclude certain records and to alter the record content adding and removing specific keys.
key_exists
Check if a key with a given name exists in the record.
key_not_exists
Check if a key does not exist in the record.
key_val_is_null
check that the value of the key is NULL.
key_val_is_not_null
check that the value of the key is NOT NULL.
key_val_eq
check that the value of the key equals the given value in the configuration.
action
action to take when a rule does not match. The available options are warn
or exit
. On warn
, a warning message is sent to the logging layer when a mismatch of the rules above is found; using exit
makes Fluent Bit abort with status code 255
.
The docker input plugin allows you to collect Docker container metrics such as memory usage and CPU consumption.
Content:
The plugin supports the following configuration parameters:
Interval_Sec
Polling interval in seconds
1
Include
A space-separated list of containers to include
Exclude
A space-separated list of containers to exclude
If you set neither Include
nor Exclude
, the plugin will try to get metrics from all the running containers.
Here is an example configuration that collects metrics from two docker instances (6bab19c3a0f9
and 14159be4ca2c
).
This configuration will produce records like below.
The docker events input plugin uses the docker API to capture server events. A complete list of possible events returned by this plugin can be found here
This plugin supports the following configuration parameters:
Unix_Path
The docker socket unix path
/var/run/docker.sock
Buffer_Size
The size of the buffer used to read docker events (in bytes)
8192
Parser
Specify the name of a parser to interpret the entry as a structured message.
None
Key
When a message is unstructured (no parser applied), it's appended as a string under the key name message.
message
Reconnect.Retry_limits
The maximum number of retries allowed. The plugin tries to reconnect with docker socket when EOF is detected.
5
Reconnect.Retry_interval
The retrying interval. Unit is second.
1
In your main configuration file append the following Input & Output sections:
The cpu input plugin, measures the CPU usage of a process or the whole system by default (considering per CPU core). It reports values in percentage unit for every interval of time set. At the moment this plugin is only available for Linux.
The following tables describes the information generated by the plugin. The keys below represent the data used by the overall system, all values associated to the keys are in a percentage unit (0 to 100%):
The CPU metrics plugin creates metrics that are log-based (I.e. JSON payload). If you are looking for Prometheus-based metrics please see the Node Exporter Metrics input plugin.
cpu_p
CPU usage of the overall system, this value is the summation of time spent on user and kernel space. The result takes in consideration the numbers of CPU cores in the system.
user_p
CPU usage in User mode, for short it means the CPU usage by user space programs. The result of this value takes in consideration the numbers of CPU cores in the system.
system_p
CPU usage in Kernel mode, for short it means the CPU usage by the Kernel. The result of this value takes in consideration the numbers of CPU cores in the system.
In addition to the keys reported in the above table, a similar content is created per CPU core. The cores are listed from 0 to N as the Kernel reports:
cpuN.p_cpu
Represents the total CPU usage by core N.
cpuN.p_user
Total CPU spent in user mode or user space programs associated to this core.
cpuN.p_system
Total CPU spent in system or kernel mode associated to this core.
The plugin supports the following configuration parameters:
Interval_Sec
Polling interval in seconds
1
Interval_NSec
Polling interval in nanoseconds
0
PID
Specify the ID (PID) of a running process in the system. By default the plugin monitors the whole system but if this option is set, it will only monitor the given process ID.
In order to get the statistics of the CPU usage of your system, you can run the plugin from the command line or through the configuration file:
As described above, the CPU input plugin gathers the overall usage every one second and flushed the information to the output on the fifth second. On this example we used the stdout plugin to demonstrate the output records. In a real use-case you may want to flush this information to some central aggregator such as Fluentd or Elasticsearch.
In your main configuration file append the following Input & Output sections:
Learn how to monitor your data pipeline with external services
A Data Pipeline represents a flow of data that goes through the inputs (sources), filters, and output (sinks). There are a couple of ways to monitor the pipeline. We recommend the following sections for a better understanding and steps to get started:
Learn how to monitor your Fluent Bit data pipelines
Fluent Bit comes with built-it features to allow you to monitor the internals of your pipeline, connect to Prometheus and Grafana, Health checks and also connectors to use external services for such purposes:
Fluent Bit comes with a built-in HTTP Server that can be used to query internal information and monitor metrics of each running plugin.
The monitoring interface can be easily integrated with Prometheus since we support it native format.
To get started, the first step is to enable the HTTP Server from the configuration file:
the above configuration snippet will instruct Fluent Bit to start it HTTP Server on TCP Port 2020 and listening on all network interfaces:
now with a simple curl command is enough to gather some information:
Note that we are sending the curl command output to the jq program which helps to make the JSON data easy to read from the terminal. Fluent Bit don't aim to do JSON pretty-printing.
Fluent Bit aims to expose useful interfaces for monitoring, as of Fluent Bit v0.14 the following end points are available:
/
Fluent Bit build information
JSON
/api/v1/uptime
Get uptime information in seconds and human readable format
JSON
/api/v1/metrics
Internal metrics per loaded plugin
JSON
/api/v1/metrics/prometheus
Internal metrics per loaded plugin ready to be consumed by a Prometheus Server
Prometheus Text 0.0.4
/api/v1/storage
Get internal metrics of the storage layer / buffered data. This option is enabled only if in the SERVICE
section the property storage.metrics
has been enabled
JSON
/api/v1/health
Fluent Bit health check result
String
/api/v2/metrics
Internal metrics per loaded plugin
/api/v2/metrics/prometheus
Internal metrics per loaded plugin ready to be consumed by a Prometheus Server
Prometheus Text 0.0.4
/api/v2/reload
JSON
The following are detailed descriptions for the metrics outputted in prometheus format by /api/v1/metrics/prometheus
.
The following definitions are key to understand:
record: a single message collected from a source, such as a single long line in a file.
chunk: Fluent Bit input plugin instances ingest log records and store them in chunks. A batch of records in a chunk are tracked together as a single unit; the Fluent Bit engine attempts to fit records into chunks of at most 2 MB, but the size can vary at runtime. Chunks are then sent to an output. An output plugin instance can either successfully send the full chunk to the destination and mark it as successful, or it can fail the chunk entirely if an unrecoverable error is encountered, or it can ask for the chunk to be retried.
fluentbit_input_bytes_total
name: the name or alias for the input instance
The number of bytes of log records that this input instance has successfully ingested
counter
bytes
fluentbit_input_records_total
name: the name or alias for the input instance
The number of log records this input has successfully ingested
counter
records
fluentbit_output_dropped_records_total
name: the name or alias for the output instance
The number of log records that have been dropped by the output. This means they met an unrecoverable error or retries expired for their chunk.
counter
records
fluentbit_output_errors_total
name: the name or alias for the output instance
The number of chunks that have faced an error (either unrecoverable or retriable). This is the number of times a chunk has failed, and does not correspond with the number of error messages you see in the Fluent Bit log output.
counter
chunks
fluentbit_output_proc_bytes_total
name: the name or alias for the output instance
The number of bytes of log records that this output instance has successfully sent. This is the total byte size of all unique chunks sent by this output. If a record is not sent due to some error, then it will not count towards this metric.
counter
bytes
fluentbit_output_proc_records_total
name: the name or alias for the output instance
The number of log records that this output instance has successfully sent. This is the total record count of all unique chunks sent by this output. If a record is not successfully sent, it does not count towards this metric.
counter
records
fluentbit_output_retried_records_total
name: the name or alias for the output instance
The number of log records that experienced a retry. Note that this is calculated at the chunk level, the count increased when an entire chunk is marked for retry. An output plugin may or may not perform multiple actions that generate many error messages when uploading a single chunk.
counter
records
fluentbit_output_retries_failed_total
name: the name or alias for the output instance
The number of times that retries expired for a chunk. Each plugin configures a Retry_Limit which applies to chunks. Once the Retry_Limit has been reached for a chunk it is discarded and this metric is incremented.
counter
chunks
fluentbit_output_retries_total
name: the name or alias for the output instance
The number of times this output instance requested a retry for a chunk.
counter
chunks
fluentbit_uptime
The number of seconds that Fluent Bit has been running.
counter
seconds
process_start_time_seconds
The Unix Epoch time stamp for when Fluent Bit started..
guage
seconds
The following are detailed descriptions for the metrics outputted in JSON format by /api/v1/storage
.
chunks.total_chunks
The total number of chunks of records that Fluent Bit is currently buffering
chunks
chunks.mem_chunks
The total number of chunks that are buffered in memory at this time. Note that chunks can be both in memory and on the file system at the same time.
chunks
chunks.fs_chunks
The total number of chunks saved to the filesystem.
chunks
chunks.fs_chunks_up
A chunk is "up" if it is in memory. So this is the count of chunks that are both in filesystem and in memory.
chunks
chunks.fs_chunks_down
The count of chunks that are "down" and thus are only in the filesystem.
chunks
input_chunks.{plugin name}.status.overlimit
Is this input instance over its configured Mem_Buf_Limit?
boolean
input_chunks.{plugin name}.status.mem_size
The size of memory that this input is consuming to buffer logs in chunks.
bytes
input_chunks.{plugin name}.status.mem_limit
The buffer memory limit (Mem_Buf_Limit) that applies to this input plugin.
bytes
input_chunks.{plugin name}.chunks.total
The current total number of chunks owned by this input instance.
chunks
input_chunks.{plugin name}.chunks.up
The current number of chunks that are "up" in memory for this input. Chunks that are "up" will also be in the filesystem layer as well if filesystem storage is enabled.
chunks
input_chunks.{plugin name}.chunks.down
The current number of chunks that are "down" in the filesystem for this input.
chunks
input_chunks.{plugin name}.chunks.busy
"Busy" chunks are chunks that are being processed/sent by outputs and are not eligible to have new data appended.
chunks
input_chunks.{plugin name}.chunks.busy_size
The sum of the byte size of each chunk which is currently marked as busy.
bytes
The following are detailed descriptions for the metrics outputted in prometheus format by /api/v2/metrics/prometheus
or /api/v2/metrics
.
The following definitions are key to understand:
record: a single message collected from a source, such as a single long line in a file.
chunk: Fluent Bit input plugin instances ingest log records and store them in chunks. A batch of records in a chunk are tracked together as a single unit; the Fluent Bit engine attempts to fit records into chunks of at most 2 MB, but the size can vary at runtime. Chunks are then sent to an output. An output plugin instance can either successfully send the full chunk to the destination and mark it as successful, or it can fail the chunk entirely if an unrecoverable error is encountered, or it can ask for the chunk to be retried.
fluentbit_input_bytes_total
name: the name or alias for the input instance
The number of bytes of log records that this input instance has successfully ingested
counter
bytes
fluentbit_input_records_total
name: the name or alias for the input instance
The number of log records this input has successfully ingested
counter
records
fluentbit_filter_bytes_total
name: the name or alias for the filter instance
The number of bytes of log records that this filter instance has successfully ingested
counter
bytes
fluentbit_filter_records_total
name: the name or alias for the filter instance
The number of log records this filter has successfully ingested
counter
records
fluentbit_filter_added_records_total
name: the name or alias for the filter instance
The number of log records that have been added by the filter. This means they added into the data pipeline.
counter
records
fluentbit_filter_dropped_records_total
name: the name or alias for the filter instance
The number of log records that have been dropped by the filter. This means they removed from the data pipeline.
counter
records
fluentbit_output_dropped_records_total
name: the name or alias for the output instance
The number of log records that have been dropped by the output. This means they met an unrecoverable error or retries expired for their chunk.
counter
records
fluentbit_output_errors_total
name: the name or alias for the output instance
The number of chunks that have faced an error (either unrecoverable or retriable). This is the number of times a chunk has failed, and does not correspond with the number of error messages you see in the Fluent Bit log output.
counter
chunks
fluentbit_output_proc_bytes_total
name: the name or alias for the output instance
The number of bytes of log records that this output instance has successfully sent. This is the total byte size of all unique chunks sent by this output. If a record is not sent due to some error, then it will not count towards this metric.
counter
bytes
fluentbit_output_proc_records_total
name: the name or alias for the output instance
The number of log records that this output instance has successfully sent. This is the total record count of all unique chunks sent by this output. If a record is not successfully sent, it does not count towards this metric.
counter
records
fluentbit_output_retried_records_total
name: the name or alias for the output instance
The number of log records that experienced a retry. Note that this is calculated at the chunk level, the count increased when an entire chunk is marked for retry. An output plugin may or may not perform multiple actions that generate many error messages when uploading a single chunk.
counter
records
fluentbit_output_retries_failed_total
name: the name or alias for the output instance
The number of times that retries expired for a chunk. Each plugin configures a Retry_Limit which applies to chunks. Once the Retry_Limit has been reached for a chunk it is discarded and this metric is incremented.
counter
chunks
fluentbit_output_retries_total
name: the name or alias for the output instance
The number of times this output instance requested a retry for a chunk.
counter
chunks
fluentbit_uptime
hostname: the hostname on running fluent-bit
The number of seconds that Fluent Bit has been running.
counter
seconds
fluentbit_process_start_time_seconds
hostname: the hostname on running fluent-bit
The Unix Epoch time stamp for when Fluent Bit started.
gauge
seconds
fluentbit_build_info
hostname: the hostname, version: the version of fluent-bit, os: OS type
Build version information. The returned value is originated from initializing the Unix Epoch time stamp of config context.
gauge
seconds
fluentbit_hot_reloaded_times
hostname: the hostname on running fluent-bit
Collect the count of hot reloaded times.
gauge
seconds
The following are detailed descriptions for the metrics which is collected by storage layer.
fluentbit_input_chunks.storage_chunks
None
The total number of chunks of records that Fluent Bit is currently buffering
gauge
chunks
fluentbit_storage_mem_chunk
None
The total number of chunks that are buffered in memory at this time. Note that chunks can be both in memory and on the file system at the same time.
gauge
chunks
fluentbit_storage_fs_chunks
None
The total number of chunks saved to the filesystem.
gauge
chunks
fluentbit_storage_fs_chunks_up
None
A chunk is "up" if it is in memory. So this is the count of chunks that are both in filesystem and in memory.
gauge
chunks
fluentbit_storage_fs_chunks_down
None
The count of chunks that are "down" and thus are only in the filesystem.
gauge
chunks
fluentbit_storage_fs_chunks_busy
None
The total number of chunks are in a busy state.
gauge
chunks
fluentbit_storage_fs_chunks_busy_bytes
None
The total bytes of chunks are in a busy state.
gauge
bytes
fluentbit_input_storage_overlimit
name: the name or alias for the input instance
Is this input instance over its configured Mem_Buf_Limit?
gauge
boolean
fluentbit_input_storage_memory_bytes
name: the name or alias for the input instance
The size of memory that this input is consuming to buffer logs in chunks.
gauge
bytes
fluentbit_input_storage_chunks
name: the name or alias for the input instance
The current total number of chunks owned by this input instance.
gauge
chunks
fluentbit_input_storage_chunks_up
name: the name or alias for the input instance
The current number of chunks that are "up" in memory for this input. Chunks that are "up" will also be in the filesystem layer as well if filesystem storage is enabled.
gauge
chunks
fluentbit_input_storage_chunks_down
name: the name or alias for the input instance
The current number of chunks that are "down" in the filesystem for this input.
gauge
chunks
fluentbit_input_storage_chunks_busy
name: the name or alias for the input instance
"Busy" chunks are chunks that are being processed/sent by outputs and are not eligible to have new data appended.
gauge
chunks
fluentbit_input_storage_chunks_busy_bytes
name: the name or alias for the input instance
The sum of the byte size of each chunk which is currently marked as busy.
gauge
bytes
fluentbit_output_upstream_total_connections
name: the name or alias for the output instance
The sum of the connection count of each output plugins.
gauge
bytes
fluentbit_output_upstream_busy_connections
name: the name or alias for the output instance
The sum of the connection count in a busy state of each output plugins.
gauge
bytes
Query the service uptime with the following command:
it should print a similar output like this:
Query internal metrics in JSON format with the following command:
it should print a similar output like this:
Query internal metrics in Prometheus Text 0.0.4 format:
this time the same metrics will be in Prometheus format instead of JSON:
By default configured plugins on runtime get an internal name in the format plugin_name.ID. For monitoring purposes, this can be confusing if many plugins of the same type were configured. To make a distinction each configured input or output section can get an alias that will be used as the parent name for the metric.
The following example set an alias to the INPUT section which is using the CPU input plugin:
Now when querying the metrics we get the aliases in place instead of the plugin name:
Fluent Bit's exposed prometheus style metrics can be leveraged to create dashboards and alerts.
The provided example dashboard is heavily inspired by Banzai Cloud's logging operator dashboard but with a few key differences such as the use of the instance
label (see why here), stacked graphs and a focus on Fluent Bit metrics.
Sample alerts are available here.
Fluent bit now supports four new configs to set up the health check.
Health_Check
enable Health check feature
Off
HC_Errors_Count
the error count to meet the unhealthy requirement, this is a sum for all output plugins in a defined HC_Period, example for output error: [2022/02/16 10:44:10] [ warn] [engine] failed to flush chunk '1-1645008245.491540684.flb', retry in 7 seconds: task_id=0, input=forward.1 > output=cloudwatch_logs.3 (out_id=3)
5
HC_Retry_Failure_Count
the retry failure count to meet the unhealthy requirement, this is a sum for all output plugins in a defined HC_Period, example for retry failure: [2022/02/16 20:11:36] [ warn] [engine] chunk '1-1645042288.260516436.flb' cannot be retried: task_id=0, input=tcp.3 > output=cloudwatch_logs.1
5
HC_Period
The time period by second to count the error and retry failure data point
60
Note: Not every error log means an error nor be counted, the errors retry failures count only on specific errors which is the example in config table description
So the feature works as: Based on the HC_Period customer setup, if the real error number is over HC_Errors_Count
or retry failure is over HC_Retry_Failure_Count
, fluent bit will be considered as unhealthy. The health endpoint will return HTTP status 500 and String error
. Otherwise it's healthy, will return HTTP status 200 and string ok
The equation is:
Note: the HC_Errors_Count and HC_Retry_Failure_Count only count for output plugins and count a sum for errors and retry failures from all output plugins which is running.
See the config example:
The command to call health endpoint
Based on the fluent bit status, the result will be:
HTTP status 200 and "ok" in response to healthy status
HTTP status 500 and "error" in response for unhealthy status
With the example config, the health status is determined by following equation:
If (HC_Errors_Count > 5) OR (HC_Retry_Failure_Count > 5) IN 5 seconds is TRUE, then it's unhealthy.
If (HC_Errors_Count > 5) OR (HC_Retry_Failure_Count > 5) IN 5 seconds is FALSE, then it's healthy.
Calyptia is a hosted service that allows you to monitor your Fluent Bit agents including data flow, metrics and configurations.
Register your Fluent Bit agent will take less than one minute, steps:
Go to the calyptia core console and sign-in
On the left menu click on settings and generate/copy your API key
In your Fluent Bit configuration file, append the following configuration section:
Make sure to replace your API key in the configuration. After a few seconds upon restart your Fluent Bit agent, the Calyptia Cloud Dashboard will list your agent. Metrics will take around 30 seconds to shows up.
If want to get in touch with Calyptia team, just send an email to hello@calyptia.com
Tap can be used to generate events or records detailing what messages pass through Fluent Bit, at what time and what filters affect them.
First, we will make sure that the container image we are going to use actually supports Fluent Bit Tap (available in Fluent Bit 2.0+):
If the --enable-chunk-trace
option is present it means Fluent Bit has support for Fluent Bit Tap but it is disabled by default, so remember to enable it with this option.
You can start fluent-bit with tracing activated from the beginning by using the trace-input
and trace-output
properties, like so:
If you see the following warning then the -Z
or --enable-chunk-tracing
option is missing:
Properties can be set for the output using the --trace-output-property
option:
With that options set the stdout plugin is now emitting traces in json_lines
format:
All three options can also be defined using the much more flexible --trace
option:
We defined the entire tap pipeline using this configuration: input=dummy.0 output=stdout output.format=json_lines
which defines the following:
input: dummy.0 (listens to the tag and/or alias dummy.0
)
output: stdout (outputs to a stdout plugin)
output.format: json_lines (sets the stdout format o json_lines
)
Tap support can also be activated and deactivated via the embedded web server:
In another terminal we can activate Tap by either using the instance id of the input; dummy.0
or its alias.
Since the alias is more predictable that is what we will use:
This response means we have activated Tap, the terminal with Fluent Bit running should now look like this:
All the records that now appear are those emitted by the activities of the dummy plugin.
This example takes the same steps but demonstrates the same mechanism works with more complicated configurations. In this example we will follow a single input of many which passes through several filters.
To make sure the window is not cluttered by the actual records generated by the input plugins we send all of it to null
.
We activate with the following 'curl' command:
Now we should start seeing output similar to the following:
When activating Tap, any plugin parameter can be given. These can be used to modify, for example, the output format, the name of the time key, the format of the date, etc.
In the next example we will use the parameter "format": "json"
to demonstrate how in Tap, stdout can be shown in Json format.
First, run Fluent Bit enabling Tap:
Next, in another terminal, we activate Tap including the output, in this case stdout, and the parameters wanted, in this case "format": "json"
:
In the first terminal, we should be seeing the output similar to the following:
This parameter shows stdout in Json format, however, as mentioned before, parameters can be passed to any plugin.
Please visit the following link for more information on other output plugins: https://docs.fluentbit.io/manual/pipeline/outputs
Here we analyze a single record from a filter event to explain the meaning of each field in detail. We chose a filter record since it includes the most details of all the record types.
The type defines at what stage the event is generated:
type=1: input record
this is the unadulterated input record
type=2: filtered record
this is a record once it has been filtered. One record is generated per filter.
type=3: pre-output record
this is the record right before it is sent for output.
Since this is a record generated by the manipulation of a record by a filter is has the type 2
.
This records the start and end of an event, it is a bit different for each event type:
type 1: when the input is received, both the start and end time.
type 2: the time when filtering is matched until it has finished processing.
type 3: the time when the input is received and when it is finally slated for output.
This is a string composed of a prefix and a number which is incremented with each record received by the input during the Tap session.
This is the plugin instance name as it is generated by Fluent Bit at runtime.
If an alias is set this field will contain the alias set for a plugin.
This is an array of all the records being sent. Since Fluent Bit handles records in chunks of multiple records and chunks are indivisible the same is done in the Tap output. Each record consists of its timestamp followed by the actual data which is a composite type of keys and values.
When the service is running we can export metrics to see the overall status of the data flow of the service. But there are other use cases where we would like to know the current status of the internals of the service, specifically to answer questions like what's the current status of the internal buffers ? , the Dump Internals feature is the answer.
Fluent Bit v1.4 introduces the Dump Internals feature that can be triggered easily from the command line triggering the CONT
Unix signal.
note: this feature is only available on Linux and BSD family operating systems
Run the following kill
command to signal Fluent Bit:
The command
pidof
aims to lookup the Process ID of Fluent Bit. You can replace the
Fluent Bit will dump the following information to the standard output interface (stdout):
The dump provides insights for every input instance configured.
Overall ingestion status of the plugin.
overlimit
mem_size
Current memory size in use by the input plugin in-memory.
mem_limit
Limit set by Mem_Buf_Limit.
When an input plugin ingest data into the engine, a Chunk is created. A Chunk can contains multiple records. Upon flush time, the engine creates a Task that contains the routes for the Chunk associated in question.
The Task dump describes the tasks associated to the input plugin:
total_tasks
Total number of active tasks associated to data generated by the input plugin.
new
Number of tasks not assigned yet to an output plugin. Tasks are in new
status for a very short period of time (most of the time this value is very low or zero).
running
Number of active tasks being processed by output plugins.
size
Amount of memory used by the Chunks being processed (Total chunks size).
The Chunks dump tells more details about all the chunks that the input plugin has generated and are still being processed.
Depending of the buffering strategy and limits imposed by configuration, some Chunks might be up
(in memory) or down
(filesystem).
total_chunks
Total number of Chunks generated by the input plugin that are still being processed by the engine.
up_chunks
Total number of Chunks that are loaded in memory.
down_chunks
Total number of Chunks that are stored in the filesystem but not loaded in memory yet.
busy_chunks
Chunks marked as busy (being flushed) or locked. Busy Chunks are immutable and likely are ready to (or being) processed.
size
Amount of bytes used by the Chunk.
size err
Number of Chunks in an error state where it size could not be retrieved.
Fluent Bit relies on a custom storage layer interface designed for hybrid buffering. The Storage Layer
entry contains a total summary of Chunks registered by Fluent Bit:
total chunks
Total number of Chunks
mem chunks
Total number of Chunks memory-based
fs chunks
Total number of Chunks filesystem based
up
Total number of filesystem chunks up in memory
down
Total number of filesystem chunks down (not loaded in memory)
The exec_wasi input plugin, allows to execute WASM program that is WASI target like as external program and collects event logs from there.
The plugin supports the following configuration parameters:
WASI_Path
The place of a WASM program file.
Parser
Specify the name of a parser to interpret the entry as a structured message.
Accessible_Paths
Specify the whilelist of paths to be able to access paths from WASM programs.
Interval_Sec
Polling interval (seconds).
Interval_NSec
Polling interval (nanosecond).
Buf_Size
Oneshot
Only run once at startup. This allows collection of data precedent to fluent-bit's startup (bool, default: false)
Here is a configuration example. in_exec_wasi can handle parser. To retrieve from structured data from WASM program, you have to create parser.conf:
Note that Time_Format
should be aligned for the format of your using timestamp. In this documents, we assume that WASM program should write JSON style strings into stdout.
Then, you can specify the above parsers.conf in the main fluent-bit configuration:
in normal operation mode allows to be configurable through or using specific arguments in the command line, while this is the ideal deployment case, there are scenarios where a more restricted configuration is required: static configuration mode.
Static configuration mode aims to include a built-in configuration in the final binary of Fluent Bit, disabling the usage of external files or flags at runtime.
The following steps assumes you are familiar with configuring Fluent Bit using text files and you have experience building it from scratch as described in the section.
In your file system prepare a specific directory that will be used as an entry point for the build system to lookup and parse the configuration files. It is mandatory that this directory contain as a minimum one configuration file called fluent-bit.conf containing the required , and sections. As an example create a new fluent-bit.conf file with the following content:
the configuration provided above will calculate CPU metrics from the running system and print them to the standard output interface.
Inside Fluent Bit source code, get into the build/ directory and run CMake appending the FLB_STATIC_CONF option pointing the configuration directory recently created, e.g:
then build it:
At this point the fluent-bit binary generated is ready to run without necessity of further configuration:
The head input plugin, allows to read events from the head of file. It's behavior is similar to the head command.
The plugin supports the following configuration parameters:
This mode is useful to get a specific line. This is an example to get CPU frequency from /proc/cpuinfo.
/proc/cpuinfo is a special file to get cpu information.
Cpu frequency is "cpu MHz : 2791.009". We can get the line with this configuration file.
Output is
In order to read the head of a file, you can run the plugin from the command line or through the configuration file:
The following example will read events from the /proc/uptime file, tag the records with the uptime name and flush them back to the stdout plugin:
In your main configuration file append the following Input & Output sections:
Note: Total interval (sec) = Interval_Sec + (Interval_Nsec / 1000000000).
e.g. 1.5s = 1s + 500000000ns
A plugin to collect Fluent Bit's own metrics
Fluent Bit exposes its to allow you to monitor the internals of your pipeline. The collected metrics can be processed similarly to those from the . They can be sent to output plugins including , or ..
Important note: Metrics collected with Node Exporter Metrics flow through a separate pipeline from logs and current filters do not operate on top of metrics.
In the following configuration file, the input plugin _node_exporter_metrics collects _metrics every 2 seconds and exposes them through our output plugin on HTTP/TCP port 2021.
You can test the expose of the metrics by using curl:
The dummy input plugin, generates dummy events. It is useful for testing, debugging, benchmarking and getting started with Fluent Bit.
The plugin supports the following configuration parameters:
You can run the plugin from the command line or through the configuration file:
In your main configuration file append the following Input & Output sections:
The elasticsearch input plugin handles both Elasticsearch and OpenSearch Bulk API requests.
The plugin supports the following configuration parameters:
Note: The Elasticsearch cluster uses "sniffing" to optimize the connections between its cluster and clients. Elasticsearch can build its cluster and dynamically generate a connection list which is called "sniffing". The hostname
will be used for sniffing information and this is handled by the sniffing endpoint.
In order to start performing the checks, you can run the plugin from the command line or through the configuration file:
From the command line you can configure Fluent Bit to handle Bulk API requests with the following options:
In your main configuration file append the following Input & Output sections:
As described above, the plugin will handle ingested Bulk API requests. For large bulk ingestions, you may have to increase buffer size with buffer_max_size and buffer_chunk_size parameters:
Note that Fluent Bit's node information is returning as Elasticsearch 8.0.0.
So, users have to specify the following configurations on their beats configurations:
For large log ingestion on these beat plugins, users might have to configure rate limiting on those beats plugins when Fluent Bit indicates that the application is exceeding the size limit for HTTP requests:
Forward is the protocol used by and to route messages between peers. This plugin implements the input service to listen for Forward messages.
The plugin supports the following configuration parameters:
In order to receive Forward messages, you can run the plugin from the command line or through the configuration file as shown in the following examples.
From the command line you can let Fluent Bit listen for Forward messages with the following options:
By default the service will listen an all interfaces (0.0.0.0) through TCP port 24224, optionally you can change this directly, e.g:
In the example the Forward messages will only arrive through network interface under 192.168.3.2 address and TCP Port 9090.
In your main configuration file append the following Input & Output sections:
Execute hot reloading or get the status of hot reloading. For more details, please refer to the .
If the plugin has been configured with , this entry will report if the plugin is over the limit or not at the moment of the dump. If it is overlimit, it will print yes
, otherwise no
.
Size of the buffer (check for allowed values)
Ingesting from beats series agents is also supported. For example, , , and are able to ingest their collected data through this plugin.
Once Fluent Bit is running, you can send some messages using the fluent-cat tool (this tool is provided by :
In we should see the following output:
Dummy
Dummy JSON record. Default: {"message":"dummy"}
Metadata
Dummy JSON metadata. Default: {}
Start_time_sec
Dummy base timestamp in seconds. Default: 0
Start_time_nsec
Dummy base timestamp in nanoseconds. Default: 0
Rate
Rate at which messages are generated expressed in how many times per second. Default: 1
Interval_sec
Set seconds of time interval at which every message is generated. If set, Rate
configuration will be ignored. Default: 0
Interval_nsec
Set nanoseconds of time interval at which every message is generated. If set, Rate
configuration will be ignored. Default: 0
Samples
If set, the events number will be limited. e.g. If Samples=3, the plugin only generates three events and stops.
Copies
Number of messages to generate each time they are generated. Defaults to 1.
Flush_on_startup
If set to true
, the first dummy event is generated at startup. Default: false
File
Absolute path to the target file, e.g: /proc/uptime
Buf_Size
Buffer size to read the file.
Interval_Sec
Polling interval (seconds).
Interval_NSec
Polling interval (nanosecond).
Add_Path
If enabled, filepath is appended to each records. Default value is false.
Key
Rename a key. Default: head.
Lines
Line number to read. If the number N is set, in_head reads first N lines like head(1) -n.
Split_line
If enabled, in_head generates key-value pair per line.
scrape_interval
The rate at which metrics are collected from the host operating system
2 seconds
scrape_on_start
Scrape metrics upon start, useful to avoid waiting for 'scrape_interval' for the first round of metrics.
false
buffer_max_size
Set the maximum size of buffer.
4M
buffer_chunk_size
Set the buffer chunk size.
512K
tag_key
Specify a key name for extracting as a tag.
NULL
meta_key
Specify a key name for meta information.
"@meta"
hostname
Specify hostname or FQDN. This parameter can be used for "sniffing" (auto-discovery of) cluster node information.
"localhost"
version
Specify Elasticsearch server version. This parameter is effective for checking a version of Elasticsearch/OpenSearch server version.
"8.0.0"
Listen
Listener network interface.
0.0.0.0
Port
TCP port to listen for incoming connections.
24224
Unix_Path
Specify the path to unix socket to receive a Forward message. If set, Listen
and Port
are ignored.
Unix_Perm
Set the permission of the unix socket file. If Unix_Path
is not set, this parameter is ignored.
Buffer_Max_Size
Specify the maximum buffer memory size used to receive a Forward message. The value must be according to the Unit Size specification.
6144000
Buffer_Chunk_Size
By default the buffer to store the incoming Forward messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by Buffer_Chunk_Size. The value must be according to the Unit Size specification.
1024000
Tag_Prefix
Prefix incoming tag with the defined value.
Tag
Override the tag of the forwarded events with the defined value.
Fluent Bit is distributed as fluent-bit package and is available for the Raspberry, specifically for Raspbian distribution, the following versions are supported:
Raspbian Bullseye (11)
Raspbian Buster (10)
The first step is to add our server GPG key to your keyring, on that way you can get our signed packages:
From the 1.9.0 and 1.8.15 releases please note that the GPG key has been updated at https://packages.fluentbit.io/fluentbit.key so ensure this new one is added.
The GPG Key fingerprint of the new key is:
The previous key is still available at https://packages.fluentbit.io/fluentbit-legacy.key and may be required to install previous versions.
The GPG Key fingerprint of the old key is:
Refer to the supported platform documentation to see which platforms are supported in each release.
On Debian and derivative systems such as Raspbian, you need to add our APT server entry to your sources lists, please add the following content at bottom of your /etc/apt/sources.list file.
Now let your system update the apt database:
We recommend upgrading your system (sudo apt-get upgrade
). This could avoid potential issues with expired certificates.
Using the following apt-get command you are able now to install the latest fluent-bit:
Now the following step is to instruct systemd to enable the service:
If you do a status check, you should see a similar output like this:
The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/syslog file.
Fluent Bit is distributed as fluent-bit package and is available for the latest (and legacy) stable Debian systems: Bookworm and Bullseye. The following architectures are supported
x86_64
aarch64 / arm64v8
A simple installation script is provided to be used for most Linux targets. This will always install the most recent version released.
This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to follow the instructions below.
The first step is to add our server GPG key to your keyring, on that way you can get our signed packages. Follow the official Debian wiki guidance: https://wiki.debian.org/DebianRepository/UseThirdParty#OpenPGP\_Key\_distribution
From the 1.9.0 and 1.8.15 releases please note that the GPG key has been updated at https://packages.fluentbit.io/fluentbit.key so ensure this new one is added.
The GPG Key fingerprint of the new key is:
The previous key is still available at https://packages.fluentbit.io/fluentbit-legacy.key and may be required to install previous versions.
The GPG Key fingerprint of the old key is:
Refer to the supported platform documentation to see which platforms are supported in each release.
On Debian, you need to add our APT server entry to your sources lists, please add the following content at bottom of your /etc/apt/sources.list file - ensure to set CODENAME
to your specific Debian release name (e.g. bookworm
for Debian 12):
Now let your system update the apt database:
We recommend upgrading your system (sudo apt-get upgrade
). This could avoid potential issues with expired certificates.
Using the following apt-get command you are able now to install the latest fluent-bit:
Now the following step is to instruct systemd to enable the service:
If you do a status check, you should see a similar output like this:
The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/syslog file.
Health input plugin allows you to check how healthy a TCP server is. It does the check by issuing a TCP connection every a certain interval of time.
The plugin supports the following configuration parameters:
Host
Name of the target host or IP address to check.
Port
TCP port where to perform the connection check.
Interval_Sec
Interval in seconds between the service checks. Default value is 1.
Internal_Nsec
Specify a nanoseconds interval for service checks, it works in conjunction with the Interval_Sec configuration key. Default value is 0.
Alert
If enabled, it will only generate messages if the target TCP service is down. By default this option is disabled.
Add_Host
If enabled, hostname is appended to each records. Default value is false.
Add_Port
If enabled, port number is appended to each records. Default value is false.
In order to start performing the checks, you can run the plugin from the command line or through the configuration file:
From the command line you can let Fluent Bit generate the checks with the following options:
In your main configuration file append the following Input & Output sections:
Once Fluent Bit is running, you will see some random values in the output interface similar to this:
The kmsg input plugin reads the Linux Kernel log buffer since the beginning, it gets every record and parse it field as priority, sequence, seconds, useconds, and message.
Prio_Level
The log level to filter. The kernel log is dropped if its priority is more than prio_level. Allowed values are 0-8. Default is 8. 8 means all logs are saved.
8
In order to start getting the Linux Kernel messages, you can run the plugin from the command line or through the configuration file:
As described above, the plugin processed all messages that the Linux Kernel reported, the output has been truncated for clarification.
In your main configuration file append the following Input & Output sections:
Collects Kubernetes Events
Kubernetes exports it events through the API server. This input plugin allows to retrieve those events as logs and get them processed through the pipeline.
db
Set a database file to keep track of recorded Kubernetes events
db.sync
Set a database sync method. values: extra, full, normal and off
normal
interval_sec
Set the polling interval for each channel.
0
interval_nsec
Set the polling interval for each channel (sub seconds: nanoseconds)
500000000
kube_url
API Server end-point
https://kubernetes.default.svc
kube_ca_file
Kubernetes TLS CA file
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
kube_ca_path
Kubernetes TLS ca path
kube_token_file
Kubernetes authorization token file.
/var/run/secrets/kubernetes.io/serviceaccount/token
kube_token_ttl
kubernetes token ttl, until it is reread from the token file.
10m
kube_request_limit
kubernetes limit parameter for events query, no limit applied when set to 0.
0
kube_retention_time
Kubernetes retention time for events.
1h
kube_namespace
Kubernetes namespace to query events from. Gets events from all namespaces by default
tls.debug
Debug level between 0 (nothing) and 4 (every detail).
0
tls.verify
Enable or disable verification of TLS peer certificate.
On
tls.vhost
Set optional TLS virtual host.
In the following configuration file, the input plugin kubernetes_events collects events every 5 seconds (default for interval_nsec) and exposes them through the standard output plugin on the console.
Event timestamp will be created from the first existing field in the following order of precendence: lastTimestamp, firstTimestamp, metadata.creationTimestamp
The mem input plugin, gathers the information about the memory and swap usage of the running system every certain interval of time and reports the total amount of memory and the amount of free available.
In order to get memory and swap usage from your system, you can run the plugin from the command line or through the configuration file:
In your main configuration file append the following Input & Output sections:
The HTTP input plugin allows you to send custom records to an HTTP endpoint.
Key
Description
default
listen
The address to listen on
0.0.0.0
port
The port for Fluent Bit to listen on
9880
tag_key
Specify the key name to overwrite a tag. If set, the tag will be overwritten by a value of the key.
buffer_max_size
Specify the maximum buffer size in KB to receive a JSON message.
4M
buffer_chunk_size
This sets the chunk size for incoming incoming JSON messages. These chunks are then stored/managed in the space available by buffer_max_size.
512K
successful_response_code
It allows to set successful response code. 200
, 201
and 204
are supported.
201
success_header
Add an HTTP header key/value pair on success. Multiple headers can be set. Example: X-Custom custom-answer
HTTP input plugin supports TLS/SSL, for more details about the properties available and general configuration, please refer to the Transport Security section.
The http input plugin allows Fluent Bit to open up an HTTP port that you can then route data to in a dynamic way. This plugin supports dynamic tags which allow you to send data with different tags through the same input. An example video and curl message can be seen below
The tag for the HTTP input plugin is set by adding the tag to the end of the request URL. This tag is then used to route the event through the system. For example, in the following curl message below the tag set is app.log**. **
because the end end path is /app_log
:
If you do not set the tag http.0
is automatically used. If you have multiple HTTP inputs then they will follow a pattern of http.N
where N is an integer representing the input.
The tag_key configuration option allows to specify the key name that will be used to overwrite a tag. The tag's value will be replaced with the value associated with the specified key. For example, setting tag_key to "custom_tag" and the log event contains a json field with the key "custom_tag" Fluent Bit will use the value of that field as the new tag for routing the event through the system.
The success_header
parameter allows to set multiple HTTP headers on success. The format is:
Fluent Bit is compatible with latest Apple macOS system on x86_64 and Apple Silicon M1 architectures. At the moment there is only an official supported package on x86_64 but you can build it from source as well by following the instructions below.
The packages can be found here: https://packages.fluentbit.io/macos/
For the next steps, you will need to have Homebrew installed in your system. If is not there, you can install it with the following command:
The Fluent Bit package on Homebrew is not officially supported, but should work for basic use cases and testing. It can be installed using:
Run the following brew command in your terminal to retrieve the dependencies:
Grab a fresh copy of the Fluent Bit source code (upstream):
Optionally, if you want to use a specific version, just checkout to the proper tag. If you want to use v1.8.13
just do:
In order to prepare the build system, we need to expose certain environment variables so Fluent Bit CMake build rules can pick the right libraries:
Change to the build/ directory inside the Fluent Bit sources:
Build Fluent Bit. Note that we are indicating to the build system "where" the final binaries and config files should be installed:
Install Fluent Bit to the directory specified above. Note that this requires root privileges due to the directory we will write information to:
The binaries and configuration examples can be located at /opt/fluent-bit/
.
Grab a fresh copy of the Fluent Bit source code (upstream):
Optionally, if you want to use a specific version, just checkout to the proper tag. If you want to use v1.9.2
just do:
In order to prepare the build system, we need to expose certain environment variables so Fluent Bit CMake build rules can pick the right libraries:
And then, creating the specific macOS SDK target (For example, specifying macOS Big Sur (11.3) SDK environment):
Change to the build/ directory inside the Fluent Bit sources:
Build the Fluent Bit macOS installer.
Then, macOS installer will be generated as:
Finally, fluent-bit-<fluent-bit version>
-(intel or apple)
.pkg will be generated.
The created installer will put binaries at /opt/fluent-bit/
.
To make the access path easier to Fluent Bit binary, in your terminal extend the PATH
variable:
Now as a simple test, try Fluent Bit by generating a simple dummy message which will be printed to the standard output interface every 1 second:
You will see an output similar to this:
To halt the process, press ctrl-c
in the terminal.
The Kafka input plugin allows subscribing to one or more Kafka topics to collect messages from an Apache Kafka service. This plugin uses the official librdkafka C library (built-in dependency).
brokers
Single or multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092.
topics
Single entry or list of topics separated by comma (,) that Fluent Bit will subscribe to.
format
Serialization format of the messages. If set to "json", the payload will be parsed as json.
none
client_id
Client id passed to librdkafka.
group_id
Group id passed to librdkafka.
fluent-bit
poll_ms
Kafka brokers polling interval in milliseconds.
500
Buffer_Max_Size
Specify the maximum size of buffer per cycle to poll kafka messages from subscribed topics. To increase throughput, specify larger size.
4M
poll_ms
Kafka brokers polling interval in milliseconds.
500
rdkafka.{property}
In order to subscribe/collect messages from Apache Kafka, you can run the plugin from the command line or through the configuration file:
The kafka plugin can read parameters through the -p argument (property), e.g:
In your main configuration file append the following Input & Output sections:
The fluent-bit source repository contains a full example of using fluent-bit to process kafka records:
The above will connect to the broker listening on kafka-broker:9092
and subscribe to the fb-source
topic, polling for new messages every 100 milliseconds.
Since the payload will be in json format, we ask the plugin to automatically parse the payload with format json
.
Every message received is then processed with kafka.lua
and sent back to the fb-sink
topic of the same broker.
The example can be executed locally with make start
in the examples/kafka_filter
directory (docker/compose is used).
Process input plugin allows you to check how healthy a process is. It does so by performing a service check at every certain interval of time specified by the user.
The Process metrics plugin creates metrics that are log-based (I.e. JSON payload). If you are looking for Prometheus-based metrics please see the Node Exporter Metrics input plugin.
The plugin supports the following configuration parameters:
In order to start performing the checks, you can run the plugin from the command line or through the configuration file:
The following example will check the health of crond process.
In your main configuration file append the following Input & Output sections:
Once Fluent Bit is running, you will see the health of process:
NGINX Exporter Metrics input plugin scrapes metrics from the NGINX stub status handler.
The plugin supports the following configuration parameters:
NGINX must be configured with a location that invokes the stub status handler. Here is an example configuration with such a location:
A much more powerful and flexible metrics API is available with NGINX Plus. A path needs to be configured in NGINX Plus first.
From the command line you can let Fluent Bit generate the checks with the following options:
To gather metrics from the command line with the NGINX Plus REST API we need to turn on the nginx_plus property, like so:
In your main configuration file append the following Input & Output sections:
And for NGINX Plus API:
You can quickly test against the NGINX server running on localhost by invoking it directly from the command line:
This documentation is copied from the nginx prometheus exporter metrics documentation: [https://github.com/nginxinc/nginx-prometheus-exporter/blob/master/README.md].
Note: for the
state
metric, the string values are converted to float64 using the following rule:"up"
->1.0
,"draining"
->2.0
,"down"
->3.0
,"unavail"
–>4.0
,"checking"
–>5.0
,"unhealthy"
->6.0
.
Note: for the
state
metric, the string values are converted to float64 using the following rule:"up"
->1.0
,"down"
->3.0
,"unavail"
–>4.0
,"checking"
–>5.0
,"unhealthy"
->6.0
.
The netif input plugin gathers network traffic information of the running system every certain interval of time, and reports them.
The Network I/O Metrics plugin creates metrics that are log-based (I.e. JSON payload). If you are looking for Prometheus-based metrics please see the Node Exporter Metrics input plugin.
The plugin supports the following configuration parameters:
In order to monitor network traffic from your system, you can run the plugin from the command line or through the configuration file:
In your main configuration file append the following Input & Output sections:
Note: Total interval (sec) = Interval_Sec + (Interval_Nsec / 1000000000).
e.g. 1.5s = 1s + 500000000ns
The Podman Metrics input plugin allows you to collect metrics from podman containers, so they can be exposed later as, for example, Prometheus counters and gauges.
The podman metrics input plugin allows Fluent Bit to gather podman container metrics. The entire procedure of collecting container list and gathering data associated with them bases on filesystem data.This plugin does not execute podman commands or send http requests to podman api - instead it reads podman configuration file and metrics exposed by /sys and /proc filesystems.
This plugin supports and automatically detects both cgroups v1 and v2.
Example Curl message for one running container
Currently supported counters are:
container_memory_usage_bytes
container_memory_max_usage_bytes
container_memory_rss
container_spec_memory_limit_bytes
container_cpu_user_seconds_total
container_cpu_usage_seconds_total
container_network_receive_bytes_total
container_network_receive_errors_total
container_network_transmit_bytes_total
container_network_transmit_errors_total
A plugin based on Prometheus Node Exporter to collect system / host level metrics
is a popular way to collect system level metrics from operating systems, such as CPU / Disk / Network / Process statistics. Fluent Bit 1.8.0 includes node exporter metrics plugin that builds off the Prometheus design to collect system level metrics without having to manage two separate processes or agents.
The initial release of Node Exporter Metrics contains a subset of collectors and metrics available from Prometheus Node Exporter and we plan to expand them over time.
Important note: Metrics collected with Node Exporter Metrics flow through a separate pipeline from logs and current filters do not operate on top of metrics.
This plugin is supported on Linux-based operating systems for the most part with macOS offering a reduced subset of metrics. The table below indicates which collector is supported on macOS.
Note: The plugin top-level scrape_interval
setting is the global default with any custom settings for individual scrape_intervals
then overriding just that specific metric scraping interval. Each collector.xxx.scrape_interval
option only overrides the interval for that specific collector and updates the associated set of provided metrics.
The overridden intervals only change the collection interval, not the interval for publishing the metrics which is taken from the global setting. For example, if the global interval is set to 5s and an override interval of 60s is used then the published metrics will be reported every 5s but for the specific collector they will stay the same for 60s until it is collected again. This feature aims to help with down-sampling when collecting metrics.
The following table describes the available collectors as part of this plugin. All of them are enabled by default and respects the original metrics name, descriptions, and types from Prometheus Exporter, so you can use your current dashboards without any compatibility problem.
note: the Version column specifies the Fluent Bit version where the collector is available.
You can test the expose of the metrics by using curl:
When deploying Fluent Bit in a container you will need to specify additional settings to ensure that Fluent Bit has access to the host operating system. The following docker command deploys Fluent Bit with specific mount paths and settings enabled to ensure that Fluent Bit can collect from the host. These are then exposed over port 2021.
If you like dashboards for monitoring, Grafana is one of the preferred options. In our Fluent Bit source code repository, we have pushed a simple **docker-compose **example. Steps:
Now open your browser in the address http://127.0.0.1:3000. When asked for the credentials to access Grafana, just use the **admin **username and admin password.
Note that by default Grafana dashboard plots the data from the last 24 hours, so just change it to Last 5 minutes to see the recent data being collected.
AWS maintains a distribution of Fluent Bit combining the latest official release with a set of Go Plugins for sending logs to AWS services. AWS and Fluent Bit are working together to rewrite their plugins for inclusion in the official Fluent Bit distribution.
Currently, the image contains Go Plugins for:
Fluent Bit includes Amazon CloudWatch Logs plugin named cloudwatch_logs
, Amazon Kinesis Firehose plugin named kinesis_firehose
and Amazon Kinesis Data Streams plugin named kinesis_streams
which are higher performance than Go plugins.
Also, Fluent Bit includes S3 output plugin named s3
.
AWS vends their container image via , and a set of highly available regional Amazon ECR repositories. For more information, see the .
The AWS for Fluent Bit image uses a custom versioning scheme because it contains multiple projects. To see what each release contains, check out the .
AWS vends SSM Public Parameters with the regional repository link for each image. These parameters can be queried by any AWS account.
To see a list of available version tags in a given region, run the following command:
To see the ECR repository URI for a given image tag in a given region, run the following:
You can use these SSM public parameters as parameters in your CloudFormation templates:
Configuration files must be flexible enough for any deployment need, but they must keep a clean and readable format.
Fluent Bit Commands extends a configuration file with specific built-in features. The list of commands available as of Fluent Bit 0.12 series are:
Configuring a logging pipeline might lead to an extensive configuration file. In order to maintain a human-readable configuration, it's suggested to split the configuration in multiple files.
The @INCLUDE command allows the configuration reader to include an external configuration file, e.g:
The above example defines the main service configuration file and also include two files to continue the configuration:
Note that despites the order of inclusion, Fluent Bit will ALWAYS respect the following order:
Service
Inputs
Filters
Outputs
The @SET command can only be used at root level of each line, meaning it cannot be used inside a section, e.g:
This page describes the main configuration file used by Fluent Bit
One of the ways to configure Fluent Bit is using a main configuration file. Fluent Bit allows to use one configuration file which works at a global scope and uses the defined previously.
The main configuration file supports four types of sections:
Service
Input
Filter
Output
In addition, it's also possible to split the main configuration file in multiple files using the feature to include external files:
Include File
The Service section defines global properties of the service, the keys available as of this version are described in the following table:
The following is an example of a SERVICE section:
An INPUT section defines a source (related to an input plugin), here we will describe the base configuration for each INPUT section. Note that each input plugin may add it own configuration keys:
The Name is mandatory and it let Fluent Bit know which input plugin should be loaded. The Tag is mandatory for all plugins except for the input forward plugin (as it provides dynamic tags).
The following is an example of an INPUT section:
A FILTER section defines a filter (related to an filter plugin), here we will describe the base configuration for each FILTER section. Note that each filter plugin may add it own configuration keys:
The Name is mandatory and it let Fluent Bit know which filter plugin should be loaded. The Match or Match_Regex is mandatory for all plugins. If both are specified, Match_Regex takes precedence.
The following is an example of an FILTER section:
The OUTPUT section specify a destination that certain records should follow after a Tag match. Currently, Fluent Bit can route up to 256 OUTPUT plugins. The configuration support the following keys:
The following is an example of an OUTPUT section:
The following configuration file example demonstrates how to collect CPU metrics and flush the results every five seconds to the standard output:
To avoid complicated long configuration files is better to split specific parts in different files and call them (include) from one main file.
Starting from Fluent Bit 0.12 the new configuration command @INCLUDE has been added and can be used in the following way:
The configuration reader will try to open the path somefile.conf, if not found, it will assume it's a relative path based on the path of the base configuration file, e.g:
Main configuration file path: /tmp/main.conf
Included file: somefile.conf
Fluent Bit will try to open somefile.conf, if it fails it will try /tmp/somefile.conf.
The @INCLUDE command only works at top-left level of the configuration line, it cannot be used inside sections.
Wildcard character (*) is supported to include multiple files, e.g:
Note files matching the wildcard character are included unsorted. If plugins ordering between files need to be preserved, the files should be included explicitly.
{property}
can be any
This plugin mimics naming convetion of docker metrics exposed by project
In the following configuration file, the input plugin _node_exporter_metrics collects _metrics every 2 seconds and exposes them through our output plugin on HTTP/TCP port 2021.
Our current plugin implements a sub-set of the available collectors in the original Prometheus Node Exporter, if you would like that we prioritize a specific collector please open a Github issue by using the following template: -
Fluent Bit supports , one way to expose this variables to Fluent Bit is through setting a Shell environment variable, the other is through the @SET command.
For scheduler and retry details, please check there:
You can also visualize Fluent Bit INPUT, FILTER, and OUTPUT configuration via
nginx_up
Gauge
Shows the status of the last metric scrape: 1
for a successful scrape and 0
for a failed one
[]
nginx_connections_accepted
Counter
Accepted client connections.
[]
nginx_connections_active
Gauge
Active client connections.
[]
nginx_connections_handled
Counter
Handled client connections.
[]
nginx_connections_reading
Gauge
Connections where NGINX is reading the request header.
[]
nginx_connections_waiting
Gauge
Idle client connections.
[]
nginx_connections_writing
Gauge
Connections where NGINX is writing the response back to the client.
[]
nginx_http_requests_total
Counter
Total http requests.
[]
nginxplus_connections_accepted
Counter
Accepted client connections
[]
nginxplus_connections_active
Gauge
Active client connections
[]
nginxplus_connections_dropped
Counter
Dropped client connections dropped
[]
nginxplus_connections_idle
Gauge
Idle client connections
[]
nginxplus_http_requests_total
Counter
Total http requests
[]
nginxplus_http_requests_current
Gauge
Current http requests
[]
nginxplus_ssl_handshakes
Counter
Successful SSL handshakes
[]
nginxplus_ssl_handshakes_failed
Counter
Failed SSL handshakes
[]
nginxplus_ssl_session_reuses
Counter
Session reuses during SSL handshake
[]
nginxplus_server_zone_processing
Gauge
Client requests that are currently being processed
server_zone
nginxplus_server_zone_requests
Counter
Total client requests
server_zone
nginxplus_server_zone_responses
Counter
Total responses sent to clients
code
(the response status code. The values are: 1xx
, 2xx
, 3xx
, 4xx
and 5xx
), server_zone
nginxplus_server_zone_discarded
Counter
Requests completed without sending a response
server_zone
nginxplus_server_zone_received
Counter
Bytes received from clients
server_zone
nginxplus_server_zone_sent
Counter
Bytes sent to clients
server_zone
nginxplus_stream_server_zone_processing
Gauge
Client connections that are currently being processed
server_zone
nginxplus_stream_server_zone_connections
Counter
Total connections
server_zone
nginxplus_stream_server_zone_sessions
Counter
Total sessions completed
code
(the response status code. The values are: 2xx
, 4xx
, and 5xx
), server_zone
nginxplus_stream_server_zone_discarded
Counter
Connections completed without creating a session
server_zone
nginxplus_stream_server_zone_received
Counter
Bytes received from clients
server_zone
nginxplus_stream_server_zone_sent
Counter
Bytes sent to clients
server_zone
nginxplus_upstream_server_state
Gauge
Current state
server
, upstream
nginxplus_upstream_server_active
Gauge
Active connections
server
, upstream
nginxplus_upstream_server_limit
Gauge
Limit for connections which corresponds to the max_conns parameter of the upstream server. Zero value means there is no limit
server
, upstream
nginxplus_upstream_server_requests
Counter
Total client requests
server
, upstream
nginxplus_upstream_server_responses
Counter
Total responses sent to clients
code
(the response status code. The values are: 1xx
, 2xx
, 3xx
, 4xx
and 5xx
), server
, upstream
nginxplus_upstream_server_sent
Counter
Bytes sent to this server
server
, upstream
nginxplus_upstream_server_received
Counter
Bytes received to this server
server
, upstream
nginxplus_upstream_server_fails
Counter
Number of unsuccessful attempts to communicate with the server
server
, upstream
nginxplus_upstream_server_unavail
Counter
How many times the server became unavailable for client requests (state 'unavail') due to the number of unsuccessful attempts reaching the max_fails threshold
server
, upstream
nginxplus_upstream_server_header_time
Gauge
Average time to get the response header from the server
server
, upstream
nginxplus_upstream_server_response_time
Gauge
Average time to get the full response from the server
server
, upstream
nginxplus_upstream_keepalives
Gauge
Idle keepalive connections
upstream
nginxplus_upstream_zombies
Gauge
Servers removed from the group but still processing active client requests
upstream
nginxplus_stream_upstream_server_state
Gauge
Current state
server
, upstream
nginxplus_stream_upstream_server_active
Gauge
Active connections
server
, upstream
nginxplus_stream_upstream_server_limit
Gauge
Limit for connections which corresponds to the max_conns parameter of the upstream server. Zero value means there is no limit
server
, upstream
nginxplus_stream_upstream_server_connections
Counter
Total number of client connections forwarded to this server
server
, upstream
nginxplus_stream_upstream_server_connect_time
Gauge
Average time to connect to the upstream server
server
, upstream
nginxplus_stream_upstream_server_first_byte_time
Gauge
Average time to receive the first byte of data
server
, upstream
nginxplus_stream_upstream_server_response_time
Gauge
Average time to receive the last byte of data
server
, upstream
nginxplus_stream_upstream_server_sent
Counter
Bytes sent to this server
server
, upstream
nginxplus_stream_upstream_server_received
Counter
Bytes received from this server
server
, upstream
nginxplus_stream_upstream_server_fails
Counter
Number of unsuccessful attempts to communicate with the server
server
, upstream
nginxplus_stream_upstream_server_unavail
Counter
How many times the server became unavailable for client connections (state 'unavail') due to the number of unsuccessful attempts reaching the max_fails threshold
server
, upstream
nginxplus_stream_upstream_zombies
Gauge
Servers removed from the group but still processing active client connections
upstream
nginxplus_location_zone_requests
Counter
Total client requests
location_zone
nginxplus_location_zone_responses
Counter
Total responses sent to clients
code
(the response status code. The values are: 1xx
, 2xx
, 3xx
, 4xx
and 5xx
), location_zone
nginxplus_location_zone_discarded
Counter
Requests completed without sending a response
location_zone
nginxplus_location_zone_received
Counter
Bytes received from clients
location_zone
nginxplus_location_zone_sent
Counter
Bytes sent to clients
location_zone
cpu
Exposes CPU statistics.
Linux,macOS
v1.8
cpufreq
Exposes CPU frequency statistics.
Linux
v1.8
diskstats
Exposes disk I/O statistics.
Linux,macOS
v1.8
filefd
Exposes file descriptor statistics from /proc/sys/fs/file-nr
.
Linux
v1.8.2
filesystem
Exposes filesystem statistics from /proc/*/mounts
.
Linux
v2.0.9
loadavg
Exposes load average.
Linux,macOS
v1.8
meminfo
Exposes memory statistics.
Linux,macOS
v1.8
netdev
Exposes network interface statistics such as bytes transferred.
Linux,macOS
v1.8.2
stat
Exposes various statistics from /proc/stat
. This includes boot time, forks, and interruptions.
Linux
v1.8
time
Exposes the current system time.
Linux
v1.8
uname
Exposes system information as provided by the uname system call.
Linux,macOS
v1.8
vmstat
Exposes statistics from /proc/vmstat
.
Linux
v1.8.2
systemd collector
Exposes statistics from systemd.
Linux
v2.1.3
thermal_zone
Expose thermal statistics from /sys/class/thermal/thermal_zone/*
Linux
v2.2.1
nvme
Exposes nvme statistics from /proc
.
Linux
v2.2.0
processes
Exposes processes statistics from /proc
.
Linux
v2.2.0
Name
Name of the input plugin.
Tag
Tag name associated to all records coming from this plugin.
Log_Level
Set the plugin's logging verbosity level. Allowed values are: off, error, warn, info, debug and trace. Defaults to the SERVICE section's Log_Level.
Name
Name of the filter plugin.
Match
A pattern to match against the tags of incoming records. It's case sensitive and support the star (*) character as a wildcard.
Match_Regex
A regular expression to match against the tags of incoming records. Use this option if you want to use the full regex syntax.
Log_Level
Set the plugin's logging verbosity level. Allowed values are: off, error, warn, info, debug and trace. Defaults to the SERVICE section's Log_Level.
Name
Name of the output plugin.
Match
A pattern to match against the tags of incoming records. It's case sensitive and support the star (*) character as a wildcard.
Match_Regex
A regular expression to match against the tags of incoming records. Use this option if you want to use the full regex syntax.
Log_Level
Set the plugin's logging verbosity level. Allowed values are: off, error, warn, info, debug and trace. Defaults to the SERVICE section's Log_Level.
Host
Name of the target host or IP address to check.
localhost
Port
Port of the target nginx service to connect to.
80
Status_URL
The URL of the Stub Status Handler.
/status
Nginx_Plus
Turn on NGINX plus mode.
true
Key
Description
Default
scrape_interval
Interval between each scrape of podman data (in seconds)
30
scrape_on_start
Should this plugin scrape podman data after it is started
false
path.config
Custom path to podman containers configuration file
/var/lib/containers/storage/overlay-containers/containers.json
path.sysfs
Custom path to sysfs subsystem directory
/sys/fs/cgroup
path.procfs
Custom path to proc subsystem directory
/proc
scrape_interval
The rate at which metrics are collected from the host operating system
5 seconds
path.procfs
The mount point used to collect process information and metrics
/proc/
path.sysfs
The path in the filesystem used to collect system metrics
/sys/
collector.cpu.scrape_interval
The rate in seconds at which cpu metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
collector.cpufreq.scrape_interval
The rate in seconds at which cpufreq metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
collector.meminfo.scrape_interval
The rate in seconds at which meminfo metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
collector.diskstats.scrape_interval
The rate in seconds at which diskstats metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
collector.filesystem.scrape_interval
The rate in seconds at which filesystem metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
collector.uname.scrape_interval
The rate in seconds at which uname metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
collector.stat.scrape_interval
The rate in seconds at which stat metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
collector.time.scrape_interval
The rate in seconds at which time metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
collector.loadavg.scrape_interval
The rate in seconds at which loadavg metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
collector.vmstat.scrape_interval
The rate in seconds at which vmstat metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
collector.thermal_zone.scrape_interval
The rate in seconds at which thermal_zone metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
collector.filefd.scrape_interval
The rate in seconds at which filefd metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
collector.nvme.scrape_interval
The rate in seconds at which nvme metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
collector.processes.scrape_interval
The rate in seconds at which system level of process metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used.
0 seconds
metrics
To specify which metrics are collected from the host operating system. These metrics depend on /proc
or /sys
fs. The actual values of metrics will be read from /proc
or /sys
when needed. cpu, cpufreq, meminfo, diskstats, filesystem, stat, loadavg, vmstat, netdev, and filefd depend on procfs. cpufreq metrics depend on sysfs.
"cpu,cpufreq,meminfo,diskstats,filesystem,uname,stat,time,loadavg,vmstat,netdev,filefd"
filesystem.ignore_mount_point_regex
Specify the regex for the mount points to prevent collection of/ignore.
`^/(dev
filesystem.ignore_filesystem_type_regex
Specify the regex for the filesystem types to prevent collection of/ignore.
`^(autofs
diskstats.ignore_device_regex
Specify the regex for the diskstats to prevent collection of/ignore.
`^(ram
systemd_service_restart_metrics
Determines if the collector will include service restart metrics
false
systemd_unit_start_time_metrics
Determines if the collector will include unit start time metrics
false
systemd_include_service_task_metrics
Determines if the collector will include service task metrics
false
systemd_include_pattern
regex to determine which units are included in the metrics produced by the systemd collector
It is not applied unless explicitly set
systemd_exclude_pattern
regex to determine which units are excluded in the metrics produced by the systemd collector
`.+\.(automount
@INCLUDE FILE
Include a configuration file
@SET KEY=VAL
Set a configuration variable
flush
Set the flush time in seconds.nanoseconds
. The engine loop uses a Flush timeout to define when is required to flush the records ingested by input plugins through the defined output plugins.
5
grace
Set the grace time in seconds
as Integer value. The engine loop uses a Grace timeout to define wait time on exit
5
daemon
Boolean value to set if Fluent Bit should run as a Daemon (background) or not. Allowed values are: yes, no, on and off. note: If you are using a Systemd based unit as the one we provide in our packages, do not turn on this option.
Off
dns.mode
Set the primary transport layer protocol used by the asynchronous DNS resolver which can be overridden on a per plugin basis
UDP
log_file
Absolute path for an optional log file. By default all logs are redirected to the standard error interface (stderr).
log_level
Set the logging verbosity level. Allowed values are: off, error, warn, info, debug and trace. Values are accumulative, e.g: if 'debug' is set, it will include error, warning, info and debug. Note that trace mode is only available if Fluent Bit was built with the WITH_TRACE option enabled.
info
parsers_file
Path for a parsers
configuration file. Multiple Parsers_File entries can be defined within the section.
plugins_file
Path for a plugins
configuration file. A plugins configuration file allows to define paths for external plugins, for an example see here.
streams_file
Path for the Stream Processor configuration file. To learn more about Stream Processing configuration go here.
http_server
Enable built-in HTTP Server
Off
http_listen
Set listening interface for HTTP Server when it's enabled
0.0.0.0
http_port
Set TCP Port for the HTTP Server
2020
coro_stack_size
Set the coroutines stack size in bytes. The value must be greater than the page size of the running system. Don't set too small value (say 4096), or coroutine threads can overrun the stack buffer. Do not change the default value of this parameter unless you know what you are doing.
24576
scheduler.cap
Set a maximum retry time in second. The property is supported from v1.8.7.
2000
scheduler.base
Set a base of exponential backoff. The property is supported from v1.8.7.
5
json.convert_nan_to_null
If enabled, NaN is converted to null when fluent-bit converts msgpack to json.
false
sp.convert_from_str_to_num
If enabled, Stream processor converts from number string to number type.
true
Proc_Name
Name of the target Process to check.
Interval_Sec
Interval in seconds between the service checks. Default value is 1.
Interval_Nsec
Specify a nanoseconds interval for service checks, it works in conjunction with the Interval_Sec configuration key. Default value is 0.
Alert
If enabled, it will only generate messages if the target process is down. By default this option is disabled.
Fd
If enabled, a number of fd is appended to each records. Default value is true.
Mem
If enabled, memory usage of the process is appended to each records. Default value is true.
Interface
Specify the network interface to monitor. e.g. eth0
Interval_Sec
Polling interval (seconds).
1
Interval_NSec
Polling interval (nanosecond).
0
Verbose
If true, gather metrics precisely.
false
Test_At_Init
If true, testing if the network interface is valid at initialization.
false
Random input plugin generate very simple random value samples using the device interface /dev/urandom, if not available it will use a unix timestamp as value.
The plugin supports the following configuration parameters:
Samples
If set, it will only generate a specific number of samples. By default this value is set to -1, which will generate unlimited samples.
Interval_Sec
Interval in seconds between samples generation. Default value is 1.
Interval_Nsec
Specify a nanoseconds interval for samples generation, it works in conjunction with the Interval_Sec configuration key. Default value is 0.
In order to start generating random samples, you can run the plugin from the command line or through the configuration file:
From the command line you can let Fluent Bit generate the samples with the following options:
In your main configuration file append the following Input & Output sections:
Once Fluent Bit is running, you will see the reports in the output interface similar to this:
Fluent Bit 1.9 includes additional metrics features to allow you to collect both logs and metrics with the same collector.
The initial release of the Prometheus Scrape metric allows you to collect metrics from a Prometheus-based endpoint at a set interval. These metrics can be routed to metric supported endpoints such as Prometheus Exporter, InfluxDB, or Prometheus Remote Write
host
The host of the prometheus metric endpoint that you want to scrape
port
The port of the prometheus metric endpoint that you want to scrape
scrape_interval
The interval to scrape metrics
10s
metrics_path
The metrics URI endpoint, that must start with a forward slash.
Note: Parameters can also be added to the path by using ?
/metrics
If an endpoint exposes Prometheus Metrics we can specify the configuration to scrape and then output the metrics. In the following example, we retrieve metrics from the HashiCorp Vault application.
Example Output
A plugin based on Process Exporter to collect process level of metrics of system metrics
Prometheus Node Exporter is a popular way to collect system level metrics from operating systems, such as CPU / Disk / Network / Process statistics. Fluent Bit 2.2 onwards includes a process exporter plugin that builds off the Prometheus design to collect process level metrics without having to manage two separate processes or agents.
The Process Exporter Metrics plugin implements collecting of the various metrics available from the 3rd party implementation of Prometheus Process Exporter and these will be expanded over time as needed.
Important note: All metrics including those collected with this plugin flow through a separate pipeline from logs and current filters do not operate on top of metrics.
This plugin is only supported on Linux based operating systems as it uses the proc
filesystem to access the relevant metrics.
macOS does not have the proc
filesystem so this plugin will not work for it.
scrape_interval
The rate at which metrics are collected.
5 seconds
path.procfs
The mount point used to collect process information and metrics. Read-only is enough
/proc/
process_include_pattern
regex to determine which names of processes are included in the metrics produced by this plugin
It is applied for all process unless explicitly set. Default is .+
.
process_exclude_pattern
regex to determine which names of processes are excluded in the metrics produced by this plugin
It is not applied unless explicitly set. Default is NULL
.
metrics
To specify which process level of metrics are collected from the host operating system. These metrics depend on /proc
fs. The actual values of metrics will be read from /proc
when needed. cpu, io, memory, state, context_switches, fd, start_time, thread_wchan, thread depend on procfs.
cpu,io,memory,state,context_switches,fd,start_time,thread_wchan,thread
cpu
Exposes CPU statistics from /proc
.
io
Exposes I/O statistics from /proc
.
memory
Exposes memory statistics from /proc
.
state
Exposes process state statistics from /proc
.
context_switches
Exposes context_switches statistics from /proc
.
fd
Exposes file descriptors statistics from /proc
.
start_time
Exposes start_time statistics from /proc
.
thread_wchan
Exposes thread_wchan from /proc
.
thread
Exposes thread statistics from /proc
.
In the following configuration file, the input plugin _process_exporter_metrics collects _metrics every 2 seconds and exposes them through our Prometheus Exporter output plugin on HTTP/TCP port 2021.
You can see the metrics by using curl:
When deploying Fluent Bit in a container you will need to specify additional settings to ensure that Fluent Bit has access to the process details. The following docker
command deploys Fluent Bit with a specific mount path for procfs
and settings enabled to ensure that Fluent Bit can collect from the host. These are then exposed over port 2021.
Development prioritises a subset of the available collectors in the the 3rd party implementation of Prometheus Process Exporter, to request others please open a Github issue by using the following template: - in_process_exporter_metrics
The serial input plugin, allows to retrieve messages/data from a Serial interface.
File
Absolute path to the device entry, e.g: /dev/ttyS0
Bitrate
The bitrate for the communication, e.g: 9600, 38400, 115200, etc
Min_Bytes
The serial interface will expect at least Min_Bytes to be available before to process the message (default: 1)
Separator
Allows to specify a separator string that's used to determinate when a message ends.
Format
Specify the format of the incoming data stream. The only option available is 'json'. Note that Format and Separator cannot be used at the same time.
In order to retrieve messages over the Serial interface, you can run the plugin from the command line or through the configuration file:
The following example loads the input serial plugin where it set a Bitrate of 9600, listen from the /dev/tnt0 interface and use the custom tag data to route the message.
The above interface (/dev/tnt0) is an emulation of the serial interface (more details at bottom), for demonstrative purposes we will write some message to the other end of the interface, in this case /dev/tnt1, e.g:
In Fluent Bit you should see an output like this:
Now using the Separator configuration, we could send multiple messages at once (run this command after starting Fluent Bit):
In your main configuration file append the following Input & Output sections:
The following content is some extra information that will allow you to emulate a serial interface on your Linux system, so you can test this Serial input plugin locally in case you don't have such interface in your computer. The following procedure has been tested on Ubuntu 15.04 running a Linux Kernel 4.0.
Download the sources
Unpack and compile
Copy the new kernel module into the kernel modules directory
Load the module
You should see new serial ports in /dev/ (ls /dev/tnt*) Give appropriate permissions to the new serial ports:
When the module is loaded, it will interconnect the following virtual interfaces:
The splunk input plugin handles Splunk HTTP HEC requests.
Key
Description
default
listen
The address to listen on
0.0.0.0
port
The port for Fluent Bit to listen on
9880
tag_key
Specify the key name to overwrite a tag. If set, the tag will be overwritten by a value of the key.
buffer_max_size
Specify the maximum buffer size in KB to receive a JSON message.
4M
buffer_chunk_size
This sets the chunk size for incoming incoming JSON messages. These chunks are then stored/managed in the space available by buffer_max_size.
512K
successful_response_code
It allows to set successful response code. 200
, 201
and 204
are supported.
201
splunk_token
Add an Splunk token for HTTP HEC.`
In order to start performing the checks, you can run the plugin from the command line or through the configuration file.
The tag for the Splunk input plugin is set by adding the tag to the end of the request URL by default. This tag is then used to route the event through the system. The default behavior of the splunk input sets the tags for the following endpoints:
/services/collector
/services/collector/event
/services/collector/raw
The requests for these endpoints are interpreted as services_collector
, services_collector_event
, and services_collector_raw
.
If you want to use the other tags for multiple instantiating input splunk plugin, you have to specify tag
property on the each of splunk plugin configurations to prevent collisions of data pipeline.
From the command line you can configure Fluent Bit to handle HTTP HEC requests with the following options:
In your main configuration file append the following Input & Output sections:
Certain configuration directives in Fluent Bit refer to unit sizes such as when defining the size of a buffer or specific limits, we can find these in plugins like Tail Input, Forward Input or in generic properties like Mem_Buf_Limit.
Starting from Fluent Bit v0.11.10, all unit sizes have been standardized across the core and plugins, the following table describes the options that can be used and what they mean:
When a suffix is not specified, it's assumed that the value given is a bytes representation.
Specifying a value of 32000, means 32000 bytes
k, K, KB, kb
Kilobyte: a unit of memory equal to 1,000 bytes.
32k means 32000 bytes.
m, M, MB, mb
Megabyte: a unit of memory equal to 1,000,000 bytes
1M means 1000000 bytes
g, G, GB, gb
Gigabyte: a unit of memory equal to 1,000,000,000 bytes
1G means 1000000000 bytes
A full feature set to access content of your records
Fluent Bit works internally with structured records and it can be composed of an unlimited number of keys and values. Values can be anything like a number, string, array, or a map.
Having a way to select a specific part of the record is critical for certain core functionalities or plugins, this feature is called Record Accessor.
consider Record Accessor a simple grammar to specify record content and other miscellaneous values.
A record accessor rule starts with the character $
. Using the structured content above as an example the following table describes how to access a record:
The following table describe some accessing rules and the expected returned value:
$log
"some message"
$labels['color']
"blue"
$labels['project']['env']
"production"
$labels['unset']
null
$labels['undefined']
If the accessor key does not exist in the record like the last example $labels['undefined']
, the operation is simply omitted, no exception will occur.
The feature is enabled on a per plugin basis, not all plugins enable this feature. As an example consider a configuration that aims to filter records using grep that only matches where labels have a color blue:
The file content to process in test.log
is the following:
Running Fluent Bit with the configuration above the output will be:
The Fluent Bit record_accessor library has a limitation in the characters that can separate template variables- only dots and commas (.
and ,
) can come after a template variable. This is because the templating library must parse the template and determine the end of a variable.
The following would be invalid templates because the two template variables are not separated by commas or dots:
$TaskID-$ECSContainerName
$TaskID/$ECSContainerName
$TaskID_$ECSContainerName
$TaskIDfooo$ECSContainerName
However, the following are valid:
$TaskID.$ECSContainerName
$TaskID.ecs_resource.$ECSContainerName
$TaskID.fooo.$ECSContainerName
And the following are valid since they only contain one template variable with nothing after it:
fooo$TaskID
fooo____$TaskID
fooo/bar$TaskID
The stdin plugin supports retrieving a message stream from the standard input interface (stdin) of the Fluent Bit process. In order to use it, specify the plugin name as the input, e.g:
If the stdin stream is closed (end-of-file), the stdin plugin will instruct Fluent Bit to exit with success (0) after flushing any pending output.
If no parser is configured for the stdin plugin, it expects valid JSON input data in one of the following formats:
A JSON object with one or more key-value pairs: { "key": "value", "key2": "value2" }
A 2-element JSON array in Fluent Bit Event format, which may be:
[TIMESTAMP, { "key": "value" }]
where TIMESTAMP is a floating point value representing a timestamp in seconds; or
from Fluent Bit v2.1.0, [[TIMESTAMP, METADATA], { "key": "value" }]
where TIMESTAMP has the same meaning as above and and METADATA is a JSON object.
Multi-line input JSON is supported.
Any input data that is not in one of the above formats will cause the plugin to log errors like:
To handle inputs in other formats, a parser must be explicitly specified in the configuration for the stdin
plugin. See parser input example for sample configuration.
The Fluent Bit event timestamp will be set from the input record if the 2-element event input is used or a custom parser configuration supplies a timestamp. Otherwise the event timestamp will be set to the timestamp at which the record is read by the stdin plugin.
A better example to demonstrate how it works will be through a Bash script that generates messages and writes them to Fluent Bit. Write the following content in a file named test.sh:
Now lets start the script and Fluent Bit:
An input event timestamp may also be supplied. Replace test.sh
with:
Re-run the sample command. Note that the timestamps output by Fluent Bit are now one day old because Fluent Bit used the input message timestamp.
Additional metadata is also supported on Fluent Bit v2.1.0 and above by replacing the timestamp with a 2-element object, e.g.:
On older Fluent Bit versions records in this format will be discarded. Fluent Bit will log:
if the log level permits.
To capture inputs in other formats, specify a parser configuration for the stdin
plugin.
For example, if you want to read raw messages line-by-line and forward them you could use a parser.conf
that captures the whole message line:
then use that in the parser
clause of the stdin plugin in the fluent-bit.conf
:
Fluent Bit will now read each line and emit a single message for each input line:
In real-world deployments it is best to use a more realistic parser that splits messages into real fields and adds appropriate tags.
The plugin supports the following configuration parameters:
Buffer_Size
16k
Parser
The name of the parser to invoke instead of the default JSON input parser
The Systemd input plugin allows to collect log messages from the Journald daemon on Linux environments.
The plugin supports the following configuration parameters:
In order to receive Systemd messages, you can run the plugin from the command line or through the configuration file:
From the command line you can let Fluent Bit listen for Systemd messages with the following options:
In the example above we are collecting all messages coming from the Docker service.
In your main configuration file append the following Input & Output sections:
Syslog input plugins allows to collect Syslog messages through a Unix socket server (UDP or TCP) or over the network using TCP or UDP.
The plugin supports the following configuration parameters:
When using Syslog input plugin, Fluent Bit requires access to the parsers.conf file, the path to this file can be specified with the option -R or through the Parsers_File key on the [SERVICE] section (more details below).
When udp or unix_udp is used, the buffer size to receive messages is configurable only through the Buffer_Chunk_Size option which defaults to 32kb.
In order to receive Syslog messages, you can run the plugin from the command line or through the configuration file:
From the command line you can let Fluent Bit listen for Forward messages with the following options:
By default the service will create and listen for Syslog messages on the unix socket /tmp/in_syslog
In your main configuration file append the following Input & Output sections:
Once Fluent Bit is running, you can send some messages using the logger tool:
The following content aims to provide configuration examples for different use cases to integrate Fluent Bit and make it listen for Syslog messages from your systems.
Put the following content in your fluent-bit.conf file:
then start Fluent Bit.
Add a new file to your rsyslog config rules called 60-fluent-bit.conf inside the directory /etc/rsyslog.d/ and add the following content:
then make sure to restart your rsyslog daemon:
Put the following content in your fluent-bit.conf file:
then start Fluent Bit.
Add a new file to your rsyslog config rules called 60-fluent-bit.conf inside the directory /etc/rsyslog.d/ and place the following content:
Make sure that the socket file is readable by rsyslog (tweak the Unix_Perm
option shown above).
The statsd input plugin allows you to receive metrics via StatsD protocol.
Content:
The plugin supports the following configuration parameters:
Here is a configuration example.
Now you can input metrics through the UDP port as follows:
Fluent Bit will produce the following records:
Set the buffer size to read data. This value is used to increase buffer size. The value must be according to the specification.
In we should see the following output:
Path
Optional path to the Systemd journal directory, if not set, the plugin will use default paths to read local-only logs.
Max_Fields
Set a maximum number of fields (keys) allowed per record.
8000
Max_Entries
When Fluent Bit starts, the Journal might have a high number of logs in the queue. In order to avoid delays and reduce memory usage, this option allows to specify the maximum number of log entries that can be processed per round. Once the limit is reached, Fluent Bit will continue processing the remaining log entries once Journald performs the notification.
5000
Systemd_Filter
Allows to perform a query over logs that contains a specific Journald key/value pairs, e.g: _SYSTEMD_UNIT=UNIT. The Systemd_Filter option can be specified multiple times in the input section to apply multiple filters as required.
Systemd_Filter_Type
Define the filter type when Systemd_Filter is specified multiple times. Allowed values are And and Or. With And a record is matched only when all of the Systemd_Filter have a match. With Or a record is matched when any of the Systemd_Filter has a match.
Or
Tag
The tag is used to route messages but on Systemd plugin there is an extra functionality: if the tag includes a star/wildcard, it will be expanded with the Systemd Unit file (_SYSTEMD_UNIT
, e.g. host.* => host.UNIT_NAME) or unknown
(e.g. host.unknown) if _SYSTEMD_UNIT
is missing.
DB
Specify the absolute path of a database file to keep track of Journald cursor.
DB.Sync
Set a default synchronization (I/O) method. values: Extra, Full, Normal, Off. This flag affects how the internal SQLite engine do synchronization to disk, for more details about each option please refer to this section. note: this option was introduced on Fluent Bit v1.4.6.
Full
Read_From_Tail
Start reading new entries. Skip entries already stored in Journald.
Off
Lowercase
Lowercase the Journald field (key).
Off
Strip_Underscores
Remove the leading underscore of the Journald field (key). For example the Journald field _PID becomes the key PID.
Off
Mode
Defines transport protocol mode: unix_udp (UDP over Unix socket), unix_tcp (TCP over Unix socket), tcp or udp
unix_udp
Listen
If Mode is set to tcp or udp, specify the network interface to bind.
0.0.0.0
Port
If Mode is set to tcp or udp, specify the TCP port to listen for incoming connections.
5140
Path
If Mode is set to unix_tcp or unix_udp, set the absolute path to the Unix socket file.
Unix_Perm
If Mode is set to unix_tcp or unix_udp, set the permission of the Unix socket file.
0644
Parser
Specify an alternative parser for the message. If Mode is set to tcp or udp then the default parser is syslog-rfc5424 otherwise syslog-rfc3164-local is used. If your syslog messages have fractional seconds set this Parser value to syslog-rfc5424 instead.
Buffer_Chunk_Size
By default the buffer to store the incoming Syslog messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by Buffer_Chunk_Size. If not set, Buffer_Chunk_Size is equal to 32000 bytes (32KB). Read considerations below when using udp or unix_udp mode.
Buffer_Max_Size
Specify the maximum buffer size to receive a Syslog message. If not set, the default size will be the value of Buffer_Chunk_Size.
Receive_Buffer_Size
Specify the maximum socket receive buffer size. If not set, the default value is OS-dependant, but generally too low to accept thousands of syslog messages per second without loss on udp or unix_udp sockets. Note that on Linux the value is capped by sysctl net.core.rmem_max
.
Source_Address_Key
Specify the key where the source address will be injected.
Listen
Listener network interface.
0.0.0.0
Port
UDP port where listening for connections
8125
The disk input plugin, gathers the information about the disk throughput of the running system every certain interval of time and reports them.
The Disk I/O metrics plugin creates metrics that are log-based (I.e. JSON payload). If you are looking for Prometheus-based metrics please see the Node Exporter Metrics input plugin.
The plugin supports the following configuration parameters:
Interval_Sec
Polling interval (seconds).
1
Interval_NSec
Polling interval (nanosecond).
0
Dev_Name
Device name to limit the target. (e.g. sda). If not set, in_disk gathers information from all of disks and partitions.
all disks
In order to get disk usage from your system, you can run the plugin from the command line or through the configuration file:
In your main configuration file append the following Input & Output sections:
Note: Total interval (sec) = Interval_Sec + (Interval_Nsec / 1000000000).
e.g. 1.5s = 1s + 500000000ns
Enable traffic through a proxy server via HTTP_PROXY environment variable
Fluent Bit supports configuring an HTTP proxy for all egress HTTP/HTTPS traffic via the HTTP_PROXY
or http_proxy
environment variable.
The format for the HTTP proxy environment variable is http://USER:PASS@HOST:PORT
, where:
USER
is the username when using basic authentication.
PASS
is the password when using basic authentication.
HOST
is the HTTP proxy hostname or IP address.
PORT
is the port the HTTP proxy is listening on.
To use an HTTP proxy with basic authentication, provide the username and password:
When no authentication is required, omit the username and password:
The HTTP_PROXY
environment variable is a standard way for setting a HTTP proxy in a containerized environment, and it is also natively supported by any application written in Go. Therefore, we follow and implement the same convention for Fluent Bit. For convenience and compatibility, the http_proxy
environment variable is also supported. When both the HTTP_PROXY
and http_proxy
environment variables are provided, HTTP_PROXY
will be preferred.
Note: The HTTP output plugin also supports configuring an HTTP proxy. This configuration continues to work, however it should not be used together with the HTTP_PROXY
or http_proxy
environment variable. This is because under the hood, the environment variable based proxy configuration is implemented by setting up a TCP connection tunnel via HTTP CONNECT. Unlike the plugin's implementation, this supports both HTTP and HTTPS egress traffic.
Not all traffic should flow through the HTTP proxy. In this case, the NO_PROXY
or no_proxy
environment variable should be used.
The format for the no proxy environment variable is a comma-separated list of hostnames or IP addresses whose traffic should not flow through the HTTP proxy.
A domain name matches itself and all its subdomains (i.e. foo.com
matches foo.com
and bar.foo.com
):
A domain with a leading .
only matches its subdomains (i.e. .foo.com
matches bar.foo.com
but not foo.com
):
One typical use case for NO_PROXY
is when running Fluent Bit in a Kubernetes environment, where we want:
All real egress traffic to flow through an HTTP proxy.
All local Kubernetes traffic to not flow through the HTTP proxy.
In this case, we can set:
For convenience and compatibility, the no_proxy
environment variable is also supported. When both the NO_PROXY
and no_proxy
environment variables are provided, NO_PROXY
will be preferred.
The collectd input plugin allows you to receive datagrams from collectd service.
The plugin supports the following configuration parameters:
Listen
Set the address to listen to
0.0.0.0
Port
Set the port to listen to
25826
TypesDB
Set the data specification file
/usr/share/collectd/types.db
Here is a basic configuration example.
With this configuration, Fluent Bit listens to 0.0.0.0:25826
, and outputs incoming datagram packets to stdout.
You must set the same types.db files that your collectd server uses. Otherwise, Fluent Bit may not be able to interpret the payload properly.
The exec input plugin, allows to execute external program and collects event logs.
WARNING: Because this plugin invokes commands via a shell, its inputs are subject to shell metacharacter substitution. Careless use of untrusted input in command arguments could lead to malicious command execution.
This plugin will not function in all the distroless production images as it needs a functional /bin/sh
which is not present. The debug images use the same binaries so even though they have a shell, there is no support for this plugin as it is compiled out.
The plugin supports the following configuration parameters:
Command
Parser
Specify the name of a parser to interpret the entry as a structured message.
Interval_Sec
Polling interval (seconds).
Interval_NSec
Polling interval (nanosecond).
Buf_Size
Oneshot
Only run once at startup. This allows collection of data precedent to fluent-bit's startup (bool, default: false)
Exit_After_Oneshot
Exit as soon as the one-shot command exits. This allows the exec plugin to be used as a wrapper for another command, sending the target command's output to any fluent-bit sink(s) then exiting. (bool, default: false)
Propagate_Exit_Code
You can run the plugin from the command line or through the configuration file:
The following example will read events from the output of ls.
In your main configuration file append the following Input & Output sections:
To use fluent-bit
with the exec
plugin to wrap another command, use the Exit_After_Oneshot
and Propagate_Exit_Code
options, e.g.:
fluent-bit
will output
then exit with exit code 1.
Translation of command exit code(s) to fluent-bit
exit code follows the usual shell rules for exit code handling. Like with a shell, there is no way to differentiate between the command exiting on a signal and the shell exiting on a signal, and no way to differentiate between normal exits with codes greater than 125 and abnormal or signal exits reported by fluent-bit or the shell. Wrapped commands should use exit codes between 0 and 125 inclusive to allow reliable identification of normal exit. If the command is a pipeline, the exit code will be the exit code of the last command in the pipeline unless overridden by shell options.
By default the exec
plugin emits one message per command output line, with a single field exec
containing the full message. Use the Parser
directive to specify the name of a parser configuration to use to process the command input.
Take great care with shell quoting and escaping when wrapping commands. A script like
can ruin your day if someone passes it the argument $(rm -rf /my/important/files; echo "deleted your stuff!")'
The above script would be safer if written with:
... but it's generally best to avoid dynamically generating the command or handling untrusted arguments to it at all.
The command to execute, passed to without any additional escaping or processing. May include pipelines, redirection, command-substitution, etc.
Size of the buffer (check for allowed values)
When exiting due to Exit_After_Oneshot, cause fluent-bit to exit with the exit code of the command exited by this plugin. Follows . (bool, default: false)
The MQTT input plugin, allows to retrieve messages/data from MQTT control packets over a TCP connection. The incoming data to receive must be a JSON map.
The plugin supports the following configuration parameters:
Listen
Listener network interface, default: 0.0.0.0
Port
TCP port where listening for connections, default: 1883
Payload_Key
Specify the key where the payload key/value will be preserved.
In order to start listening for MQTT messages, you can run the plugin from the command line or through the configuration file:
Since the MQTT input plugin let Fluent Bit behave as a server, we need to dispatch some messages using some MQTT client, in the following example mosquitto tool is being used for the purpose:
The following command line will send a message to the MQTT input plugin:
In your main configuration file append the following Input & Output sections: