In 2014, the Fluentd team at Treasure Data was forecasting the need for a lightweight log processor for constraint environments like embedded Linux and gateways. The project aimed to be part of the Fluentd ecosystem. At that moment, Eduardo Silva created Fluent Bit, a new open source solution, written from scratch and available under the terms of the Apache License v2.0.

After the project matured, it gained traction for normal Linux systems. With the new containerized world, the Cloud Native community asked to extend the project scope to support more sources, filters, and destinations. Not long after, Fluent Bit became one of the preferred solutions to solve the logging challenges in Cloud environments.

Fluent Bit is a fast and lightweight telemetry agent for logs, metrics, and traces for Linux, macOS, Windows, and BSD family operating systems. Fluent Bit has been made with a strong focus on performance to allow the collection and processing of telemetry data from different sources without complexity.

Features

• High performance: High throughput with low resources consumption

• Data parsing

  • Convert your unstructured messages using our parsers: , , and

• Metrics support: Prometheus and OpenTelemetry compatible

• Reliability and data integrity

  • handling

  • in memory and file system

• Networking

  • Security: Built-in TLS/SSL support

  • Asynchronous I/O

• Pluggable architecture and : Inputs, Filters and Outputs:

  • Connect nearly any source to nearly any destination using preexisting plugins

  • Extensibility:

    • Write input, filter, or output plugins in the C language
• : Expose internal metrics over HTTP in JSON and format

• : Perform data selection and transformation using simple SQL queries

  • Create new streams of data using query results

  • Aggregation windows

  • Data analysis and prediction: Timeseries forecasting

• Portable: Runs on Linux, macOS, Windows and BSD systems

Fluent Bit, Fluentd and CNCF

is a graduated sub-project under the umbrella of . Fluent Bit is licensed under the terms of the .

Fluent Bit was originally created by and is now sponsored by . As a CNCF-hosted project, it is a fully vendor-neutral and community-driven project.

Telemetry data processing can be complex, especially at scale. That's why was created. Fluentd is more than a simple tool, it's grown into a fullscale ecosystem that contains SDKs for different languages and subprojects like .

Here, we describe the relationship between the and open source projects.

Both projects are:

• Licensed under the terms of Apache License v2.0.

• Graduated hosted projects by the .

When processes data, it uses the system memory (heap) as a primary and temporary place to store the record logs before they get delivered. The records are processed in this private memory area.

Buffering is the ability to store the records, and continue storing incoming data while previous data is processed and delivered. Buffering in memory is the fastest mechanism, but there are scenarios requiring special strategies to deal with , data safety, or to reduce memory consumption by the service in constrained environments.

Network failures or latency in third party service is common. When data can't be delivered fast enough and new data to process arrives, the system can face backpressure.

Fluent Bit buffering strategies are designed to solve problems associated with backpressure and general delivery failures. Fluent Bit offers a primary buffering mechanism in memory and an optional secondary one using the file system. With this hybrid solution you can accommodate any use case safely and keep a high performance while processing your data.

These mechanisms aren't mutually exclusive. When data is ready to be processed or delivered it's always be in memory, while other data in the queue might be in the file system until is ready to be processed and moved up to memory.

To learn more about the buffering configuration in Fluent Bit, see .

The phase in the pipeline aims to provide a unified and persistent mechanism to store your data, using the primary in-memory model or the file system-based mode.

The buffer phase contains the data in an immutable state, meaning that no other filter can be applied.

Buffered data uses the Fluent Bit internal binary representation, which isn't raw text.

Fluent Bit offers a buffering mechanism in the file system that acts as a backup system to avoid data loss in case of system failures.

You can download the most recent stable or development source code.

Stable

For production systems, it's strongly suggested that you get the latest stable release of the source code in either zip file or tarball file format from GitHub using the following link pattern:

For example, for version 1.8.12 the link is:

Multiline parsers are used to combine logs that span multiple events into a single, cohesive message. This is particularly useful for handling stack traces, error logs, or any log entry that contains multiple lines of information.

In YAML configuration, the syntax for defining multiline parsers differs slightly from the classic configuration format introducing minor breaking changes, specifically on how the rules are defined.

Below is an example demonstrating how to define a multiline parser directly in the main configuration file, as well as how to include additional definitions from external files:

The example above defines a multiline parser named multiline-regex-test that uses regular expressions to handle multi-event logs. The parser contains two rules: the first rule transitions from start_state to cont when a matching log entry is detected, and the second rule continues to match subsequent lines.

For more detailed information on configuring multiline parsers, including advanced options and use cases, please refer to the Configuring Multiline Parsers section.

Fluent Bit has very low CPU and memory consumption. It's compatible with most x86-, x86_64-, arm32v7-, and arm64v8-based platforms.

The build process requires the following components:

• Compiler: GCC or clang

• CMake

• Flex and Bison: Required for Stream Processor or Record Accessor

• Libyaml development headers and libraries

Core has no other dependencies. Some features depend on third-party components. For example, output plugins with special backend libraries like Kafka include those libraries in the main source code repository.

Fluent Bit is supported on Linux on IBM Z(s390x), but the WASM and LUA filter plugins aren't.

Learn how to install Fluent Bit and the AWS output plugins on Amazon Linux 2 using AWS Systems Manager.

Parsers enable Fluent Bit components to transform unstructured data into a structured internal representation. You can define parsers either directly in the main configuration file or in separate external files for better organization.

This page provides a general overview of how to declare parsers.

The main section name is parsers, and it allows you to define a list of parser configurations. The following example demonstrates how to set up two simple parsers:

parsers:
  - name: json
    format: json

  - name: docker
    format: json
    time_key: time
    time_format: "%Y-%m-%dT%H:%M:%S.%L"
    time_keep: true

You can define multiple parsers sections, either within the main configuration file or distributed across included files.

For more detailed information on parser options and advanced configurations, please refer to the Configuring Parsers section.

In production environments you need full control of the data you're collecting. Filtering lets you alter the collected data before delivering it to a destination.

Filtering is implemented through plugins. Each available filter can be used to match, exclude, or enrich your logs with specific metadata.

Fluent Bit support many filters. A common use case for filtering is Kubernetes deployments. Every pod log needs the proper metadata associated with it.

Like input plugins, filters run in an instance context, which has its own independent configuration. Configuration keys are often called properties.

For more details about the Filters available and their usage, see Filters.

Fluent Bit Sandbox - sign-up required

The following are labs that can run in your browser however require email sign-up

• Fluent Bit 101 Sandbox - Getting Started with configuration and routing

Open Source Labs - environment required

The following are open source labs where you will need to spin up resources to run through the lab in details

O11y Workshops by Chronosphere

These workshops, open source, provided by Chronosphere can be found here: . The OSS repository can be found here:

The cards below include links to each of the labs in the workshop

Logging with Fluent Bit and Amazon OpenSearch workshop by Amazon

This workshop by Amazon goes through common Kubernetes logging patterns and routing data to OpenSearch and visualizing with OpenSearch dashboards

The output interface lets you define destinations for your data. Common destinations are remote services, local file systems, or other standard interfaces. Outputs are implemented as plugins.

When an output plugin is loaded, an internal instance is created. Every instance has its own independent configuration. Configuration keys are often called properties.

Every output plugin has its own documentation section specifying how it can be used and what properties are available.

For more details, see Output Plugins.

Install Fluent Bit in your embedded Linux system.

Install

To install, select Fluent Bit in your defconfig. See the Config.in file for all configuration options.

BR2_PACKAGE_FLUENT_BIT=y

Run

The default configuration file is written to:

Fluent Bit is started by the S99fluent-bit script.

Support

All configurations with a toolchain that supports threads and dynamic library linking are supported.

The includes section allows you to specify additional YAML configuration files to be merged into the current configuration. These files are identified as a list of filenames and can include relative or absolute paths. If no absolute path is provided, the file is assumed to be located in a directory relative to the file that references it.

This feature is useful for organizing complex configurations into smaller, manageable files and including them as needed.

Usage

Below is an example demonstrating how to include additional YAML files using relative path references. This is the file system path structure

├── fluent-bit.yaml
├── inclusion-1.yaml
└── subdir
    └── inclusion-2.yaml

The content of fluent-bit.yaml

Key Points

• Relative Paths: If a path is not specified as absolute, it will be treated as relative to the file that includes it.

• Organized Configurations: Using the includes section helps keep your configuration modular and easier to maintain.

note: Ensure that the included files are formatted correctly and contain valid YAML configurations for seamless integration.

Fluent Bit provides input plugins to gather information from different sources. Some plugins collect data from log files, while others can gather metrics information from the operating system. There are many plugins to suit different needs.

When an input plugin loads, an internal instance is created. Each instance has its own independent configuration. Configuration keys are often called properties.

Every input plugin has its own documentation section that specifies how to use it and what properties are available.

For more details, see Input Plugins.

A Data Pipeline represents a flow of data that goes through the inputs (sources), filters, and output (sinks). There are a couple of ways to monitor the pipeline. We recommend the following sections for a better understanding and steps to get started:

• HTTP Server: JSON and Prometheus Exporter-style metrics

• Grafana Dashboards and Alerts

• Health Checks

Fluent Bit is an open source telemetry agent specifically designed to efficiently handle the challenges of collecting and processing telemetry data across a wide range of environments, from constrained systems to complex cloud infrastructures. Managing telemetry data from various sources and formats can be a constant challenge, particularly when performance is a critical factor.

Rather than serving as a drop-in replacement, Fluent Bit enhances the observability strategy for your infrastructure by adapting and optimizing your existing logging layer, and adding metrics and traces processing. Fluent Bit supports a vendor-neutral approach, seamlessly integrating with other ecosystems such as Prometheus and OpenTelemetry. Trusted by major cloud providers, banks, and companies in need of a ready-to-use telemetry agent solution, Fluent Bit effectively manages diverse data sources and formats while maintaining optimal performance and keeping resource consumption low.

Fluent Bit can be deployed as an edge agent for localized telemetry data handling or utilized as a central aggregator/collector for managing telemetry data across multiple sources and environments.

Fluent Bit is distributed as the fluent-bit package and is available for long-term support releases of Ubuntu. The latest officially supported version is Noble Numbat (24.04).

Single line install

An installation script is provided for most Linux targets. This will always install the most recent version released.

This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to use the following instructions.

The Upstream Servers section defines a group of endpoints, referred to as nodes, which are used by output plugins to distribute data in a round-robin fashion. This is particularly useful for plugins that require load balancing when sending data. Examples of plugins that support this capability include and .

In YAML, this section is named upstream_servers and requires specifying a name for the group and a list of nodes. Below is an example that defines two upstream server groups: forward-balancing and forward-balancing-2:

Fluent Bit supports the usage of environment variables in any value associated to a key when using a configuration file.

The variables are case sensitive and can be used in the following format:

When Fluent Bit starts, the configuration reader will detect any request for ${MY_VARIABLE} and will try to resolve its value.

When Fluent Bit is running under systemd (using the official packages), environment variables can be set in the following files:

• /etc/default/fluent-bit (Debian based system)

source code provides BitBake recipes to configure, build, and package the software for a Yocto-based image. Specific steps in the usage of these recipes in your Yocto environment (Poky) is out of the scope of this documentation.

Fluent Bit distributes two main recipes, one for testing/dev purposes and one with the latest stable release.

You might need to estimate how much memory Fluent Bit could be using in scenarios like containerized environments where memory limits are essential.

To make an estimate, in-use input plugins must set the Mem_Buf_Limitoption. Learn more about it in .

Estimating

Input plugins append data independently. To make an estimation, impose a limit with the Mem_Buf_Limit option. If the limit was set to 10MB

Dealing with raw strings or unstructured messages is difficult. Having a structure makes data more usable. Set a structure to the incoming data by using input plugins as data is collected:

The parser converts unstructured data to structured data. As an example, consider the following Apache (HTTP Server) log entry:

192.168.2.20 - - [28/Jul/2006:10:27:10 -0300] "GET /cgi-bin/try/ HTTP/1.0" 200 3395

This log line is a raw string without format. Structuring the log makes it easier to process the data later. If the regular expression parser is used, the log entry could be converted to:

{
  "host":    "192.168.2.20",
  "user":    "-",
  "method":  "GET",
  "path":    "/cgi-bin/try/",
  "code":    "200",
  "size":    "3395",
  "referer": "",
  "agent":   ""
 }

Parsers are fully configurable and are independently and optionally handled by each input plugin. For more details, see Parsers.

Routing is a core feature that lets you route your data through filters and then to one or multiple destinations. The router relies on the concept of Tags and Matching rules.

There are two important concepts in Routing:

• Tag

• Match

When data is generated by an input plugin, it comes with a Tag. A Tag is a human-readable indicator that helps to identify the data source. Tags are usually configured manually.

To define where to route data, specify a Match rule in the output configuration.

Consider the following configuration example that delivers CPU metrics to an Elasticsearch database and Memory (mem) metrics to the standard output interface:

Routing reads the Input Tag and the Output Match rules. If data has a Tag that doesn't match at routing time, the data is deleted.

Routing with Wildcard

Routing is flexible enough to support wildcards in the Match pattern. The following example defines a common destination for both sources of data:

The match rule is set to my_*, which matches any Tag starting with my_*.

Routing with Regex

Routing also provides support for regular expressions with the Match_Regex pattern, allowing for more complex and precise matching criteria. The following example demonstrates how to route data from sources based on a regular expression:

In this configuration, the Match_regex rule is set to .*_sensor_[AB]. This regular expression matches any Tag that ends with _sensor_A or _sensor_B, regardless of what precedes it. This approach provides a more flexible and powerful way to handle different source tags with a single routing rule.

Before You Get Started

Fluent Bit traditionally offered a classic configuration mode, a custom configuration format that we are gradually phasing out. While classic mode has served well for many years, it has several limitations. Its basic design only supports grouping sections with key-value pairs and lacks the ability to handle sub-sections or complex data structures like lists.

YAML, now a mainstream configuration format, has become essential in a cloud ecosystem where everything is configured this way. To minimize friction and provide a more intuitive experience for creating data pipelines, we strongly encourage users to transition to YAML. The YAML format enables features, such as processors, that are not possible to configure in classic mode.

As of Fluent Bit v3.2, you can configure everything in YAML.

List of Available Sections

Configuring Fluent Bit with YAML introduces the following root-level sections:

Section Documentation

To access detailed configuration guides for each section, use the following links:

• • Overview of global settings, configuration options, and examples.

• • Detailed guide on defining parsers and supported formats.

The most secure option is to create the repositories according to the instructions for your specific OS.

An installation script is provided for use with most Linux targets. This will by default install the most recent version released.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

This is a helper and should always be validated prior to use.

GPG key updates

For the 1.9.0 and 1.8.15 releases and later, the GPG key has been updated. Ensure the new key is added.

The GPG Key fingerprint of the new key is:

C3C0 A285 34B9 293E AF51  FABD 9F9D DC08 3888 C1CD
Fluentbit releases (Releases signing key) <[email protected]>

The previous key is still available and might be required to install previous versions.

The GPG Key fingerprint of the old key is:

Refer to the to see which platforms are supported in each release.

Migration to Fluent Bit

For version 1.9 and later, td-agent-bit is a deprecated package and is removed after 1.9.9. The correct package name to use now is fluent-bit.

Certain configuration directives in Fluent Bit refer to unit sizes such as when defining the size of a buffer or specific limits, we can find these in plugins like Tail Input, Forward Input or in generic properties like Mem_Buf_Limit.

Starting from Fluent Bit v0.11.10, all unit sizes have been standardized across the core and plugins, the following table describes the options that can be used and what they mean:

Install on Amazon Linux

Fluent Bit is distributed as the fluent-bit package and is available for the latest Amazon Linux 2 and Amazon Linux 2023. The following architectures are supported

• x86_64

• aarch64 / arm64v8

Amazon Linux 2022 is no longer supported.

Single line install

Fluent Bit provides an installation script to use for most Linux targets. This will always install the most recently released version.

This is a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to use the following instructions:

Configure Yum

The fluent-bit is provided through a Yum repository. To add the repository reference to your system, add a new file called fluent-bit.repo in /etc/yum.repos.d/ with the following content:

Amazon Linux 2

Amazon Linux 2023

You should always enable gpgcheck for security reasons. All Fluent Bit packages are signed.

Updated key from March 2022

For the 1.9.0 and 1.8.15 and later releases, the . Ensure this new one is added.

The GPG Key fingerprint of the new key is:

The previous key is and might be required to install previous versions.

The GPG Key fingerprint of the old key is:

Refer to the to see which platforms are supported in each release.

Install

1. After your repository is configured, run the following command to install it:

2. Instruct systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default Fluent Bit configuration collect metrics of CPU usage and sends the records to the standard output. You can see the outgoing data in your /var/log/messages file.

While Fluent Bit comes with a variety of built-in plugins, it also supports loading external plugins at runtime. This feature is especially useful for loading Go or Wasm plugins that are built as shared object files (.so). Fluent Bit's YAML configuration provides two ways to load these external plugins:

1. Inline YAML Section

You can specify external plugins directly within your main YAML configuration file using the plugins section. Here’s an example:

plugins:
  - /path/to/out_gstdout.so

service:
  log_level: info

pipeline:
  inputs:
    - name: random

  outputs:
    - name: gstdout
      match: '*'

2. YAML Plugins File Included via plugins_file Option

Alternatively, you can load external plugins from a separate YAML file by specifying the plugins_file option in the service section. Here’s how to configure this:

In this setup, the extra_plugins.yaml file might contain the following plugins section:

Key Points

• Built-in vs. External: Fluent Bit comes with many built-in plugins, but you can load external plugins at runtime to extend the tool’s functionality.

• Loading Mechanism: External plugins must be shared object files (.so). You can define them inline in the main YAML configuration or include them from a separate YAML file for better modularity.

Fluent Bit in normal operation mode is configurable through text files or using specific arguments in the command line. Although this is the ideal deployment case, there are scenarios where a more restricted configuration is required. Static configuration mode restricts configuration ability.

Static configuration mode includes a built-in configuration in the final binary of Fluent Bit, disabling the usage of external files or flags at runtime.

Get started

Requirements

The following steps assume you are familiar with configuring Fluent Bit using text files and you have experience building it from scratch as described in .

Configuration Directory

In your file system, prepare a specific directory that will be used as an entry point for the build system to lookup and parse the configuration files. This directory must contain a minimum of one configuration file called fluent-bit.conf containing the required , , and sections.

As an example, create a new fluent-bit.conf file with the following content:

This configuration calculates CPU metrics from the running system and prints them to the standard output interface.

Build with custom configuration

1. Go to the Fluent Bit source code build directory:

2. Run CMake, appending the FLB_STATIC_CONF option pointing to the configuration directory recently created:

3. Build Fluent Bit:

The generated fluent-bit binary is ready to run without additional configuration:

The env section allows you to define environment variables directly within the configuration file. These variables can then be used to dynamically replace values throughout your configuration using the ${VARIABLE_NAME} syntax.

Values set in the env section are case-sensitive. However, as a best practice, we recommend using uppercase names for environment variables. The example below defines two variables, FLUSH_INTERVAL and STDOUT_FMT, which can be accessed in the configuration using ${FLUSH_INTERVAL} and ${STDOUT_FMT}:

env:
  FLUSH_INTERVAL: 1
  STDOUT_FMT: 'json_lines'

service:
  flush: ${FLUSH_INTERVAL}
  log_level: info

pipeline:
  inputs:
    - name: random

  outputs:
    - name: stdout
      match: '*'
      format: ${STDOUT_FMT}

Predefined Variables

Fluent Bit provides a set of predefined environment variables that can be used in your configuration:

External Variables

In addition to variables defined in the configuration file or the predefined ones, Fluent Bit can access system environment variables set in the user space. These external variables can be referenced in the configuration using the same ${VARIABLE_NAME} pattern.

For example, to set the FLUSH_INTERVAL system environment variable to 2 and use it in your configuration:

In the configuration file, you can then access this value as follows:

This approach allows you to easily manage and override configuration values using environment variables, providing flexibility in various deployment environments.

Fluent Bit is designed for high performance and minimal resource usage. Depending on your use case, you can optimize further using specific configuration options to achieve faster performance or reduce resource consumption.

Reading Files with Tail

The Tail input plugin is used to read data from files on the filesystem. By default, it uses a small memory buffer of 32KB per monitored file. While this is sufficient for most generic use cases and helps keep memory usage low when monitoring many files, there are scenarios where you may want to increase performance by using more memory.

If your files are typically larger than 32KB, consider increasing the buffer size to speed up file reading. For example, you can experiment with a buffer size of 128KB:

By increasing the buffer size, Fluent Bit will make fewer system calls (read(2)) to read the data, reducing CPU usage and improving performance.

Fluent Bit and SIMD for JSON Encoding

Starting in Fluent Bit v3.2, performance improvements have been introduced for JSON encoding. Plugins that convert logs from Fluent Bit’s internal binary representation to JSON can now do so up to 30% faster using SIMD (Single Instruction, Multiple Data) optimizations.

Enabling SIMD Support

Ensure that your Fluent Bit binary is built with SIMD support. This feature is available for architectures such as x86_64, amd64, aarch64, and arm64. As of now, SIMD is only enabled by default in Fluent Bit container images.

You can check if SIMD is enabled by looking for the following log entry when Fluent Bit starts:

Look for the simd entry, which will indicate the SIMD support type, such as SSE2, NEON, or none.

If your Fluent Bit binary was not built with SIMD enabled and you are using a supported platform, you can build Fluent Bit from source using the CMake option -DFLB_SIMD=On.

Run input plugins in threaded mode

By default, most of input plugins runs in the same system thread than the main event loop, however by configuration you can instruct them to run in a separate thread which will allow you to take advantage of other CPU cores in your system.

To run an input plugin in threaded mode, just add threaded: true as in the example below:

Fluent Bit supports the reloading feature when enabled in the configuration file or on the command line with -Y or --enable-hot-reload option.

Hot reloading is supported on Linux, macOS, and Windows operating systems.

Update the configuration

To get started with reloading over HTTP, enable the HTTP Server in the configuration file:

[SERVICE]
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_PORT    2020
    Hot_Reload   On
...

How to reload

After updating the configuration, use one of the following methods to perform a hot reload:

HTTP

Use the following HTTP endpoints to perform a hot reload:

• PUT /api/v2/reload

• POST /api/v2/reload

For using curl to reload Fluent Bit, users must specify an empty request body as:

Signal

Hot reloading can be used with SIGHUP.

SIGHUP signal isn't supported on Windows.

Confirm a reload

Use one of the following methods to confirm the reload occurred.

HTTP

Obtain a count of hot reload using the HTTP endpoint:

• GET /api/v2/reload

The endpoint returns hot_reload_count as follows:

The default value of the counter is 0.

Fluent Bit is distributed as the fluent-bit package and is available for the Raspberry, specifically for Raspbian distribution. The following versions are supported:

• Raspbian Bullseye (11)

• Raspbian Buster (10)

Server GPG key

The first step is to add the Fluent Bit server GPG key to your keyring so you can get FLuent Bit signed packages:

Updated key from March 2022

For the 1.9.0 and 1.8.15 and later releases, the . Ensure this new one is added.

The GPG Key fingerprint of the new key is:

The previous key is and might be required to install previous versions.

The GPG Key fingerprint of the old key is:

Refer to the to see which platforms are supported in each release.

Update your sources lists

On Debian and derivative systems such as Raspbian, you need to add the Fluent Bit APT server entry to your sources lists.

Add the following content at bottom of your /etc/apt/sources.list file.

Raspbian 11 (Bullseye)

Raspbian 10 (Buster)

Update your repositories database

Now let your system update the apt database:

Install Fluent Bit

1. Use the following apt-get command to install the latest Fluent Bit:

2. Instruct systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default configuration of Fluent Bit collects metrics for CPU usage and sends the records to the standard output. You can see the outgoing data in your /var/log/syslog file.

Configuration files must be flexible enough for any deployment need, but they must keep a clean and readable format.

Fluent Bit Commands extends a configuration file with specific built-in features. The list of commands available as of Fluent Bit 0.12 series are:

@INCLUDE Command

Configuring a logging pipeline might lead to an extensive configuration file. In order to maintain a human-readable configuration, it's suggested to split the configuration in multiple files.

The @INCLUDE command allows the configuration reader to include an external configuration file, e.g:

The above example defines the main service configuration file and also include two files to continue the configuration:

inputs.conf

outputs.conf

Note that despites the order of inclusion, Fluent Bit will ALWAYS respect the following order:

• Service

• Inputs

• Filters

• Outputs

@SET Command

Fluent Bit supports , one way to expose this variables to Fluent Bit is through setting a Shell environment variable, the other is through the @SET command.

The @SET command can only be used at root level of each line, meaning it cannot be used inside a section, e.g:

The collectd input plugin allows you to receive datagrams from collectd service.

Configuration Parameters

The plugin supports the following configuration parameters:

Configuration Examples

Here is a basic configuration example.

With this configuration, Fluent Bit listens to 0.0.0.0:25826, and outputs incoming datagram packets to stdout.

You must set the same types.db files that your collectd server uses. Otherwise, Fluent Bit may not be able to interpret the payload properly.

Fluent Bit has one event loop to handle critical operations, like managing timers, receiving internal messages, scheduling flushes, and handling retries. This event loop runs in the main Fluent Bit thread.

To free up resources in the main thread, you can configure inputs and outputs to run in their own self-contained threads. However, inputs and outputs implement multithreading in distinct ways: inputs can run in threaded mode, and outputs can use one or more workers.

Threading also affects certain processes related to inputs and outputs. For example, filters always run in the main thread, but processors run in the self-contained threads of their respective inputs or outputs, if applicable.

Inputs

When inputs collect telemetry data, they can either perform this process inside the main Fluent Bit thread or inside a separate dedicated thread. You can configure this behavior by enabling or disabling the threaded setting.

All inputs are capable of running in threaded mode, but certain inputs always run in threaded mode regardless of configuration. These always-threaded inputs are:

Inputs aren't internally aware of multithreading. If an input runs in threaded mode, Fluent Bit manages the logistics of that input's thread.

Outputs

When outputs flush data, they can either perform this operation inside Fluent Bit's main thread or inside a separate dedicated thread called a worker. Each output can have one or more workers running in parallel, and each worker can handle multiple concurrent flushes. You can configure this behavior by changing the value of the workers setting.

All outputs are capable of running in multiple workers, and each output has a default value of 0, 1, or 2 workers. However, even if an output uses workers by default, you can safely reduce the number of workers below the default or disable workers entirely.

Configuration Parameters

The plugin supports the following configuration parameters:

If you set neither Include nor Exclude, the plugin will try to get metrics from all the running containers.

Configuration File

Here is an example configuration that collects metrics from two docker instances (6bab19c3a0f9 and 14159be4ca2c).

This configuration will produce records like below.

The disk input plugin, gathers the information about the disk throughput of the running system every certain interval of time and reports them.

The Disk I/O metrics plugin creates metrics that are log-based, such as JSON payload. For Prometheus-based metrics, see the Node Exporter Metrics input plugin.

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

In order to get disk usage from your system, you can run the plugin from the command line or through the configuration file:

Command Line

Configuration File

In your main configuration file append the following Input & Output sections:

Note: Total interval (sec) = Interval_Sec + (Interval_Nsec / 1000000000).

e.g. 1.5s = 1s + 500000000ns

Before diving into you might want to get acquainted with some of the key concepts of the service. This document provides an introduction to those concepts and common terminology. Reading this document will help you gain a more general understanding of the following topics:

• Event or Record

• Filtering

• Tag

Fluent Bit is distributed as the fluent-bit package and is available for the latest stable CentOS system.

The following architectures are supported

• x86_64

• aarch64

AWS maintains a distribution of Fluent Bit that combines the latest official release with a set of Go Plugins for sending logs to AWS services. AWS and Fluent Bit are working together to rewrite their plugins for inclusion in the official Fluent Bit distribution.

Plugins

The image contains Go Plugins for:

• Amazon CloudWatch as cloudwatch_logs. See the

Fluent Bit might optionally use a configuration file to define how the service will behave.

Before proceeding we need to understand how the configuration schema works.

The schema is defined by three concepts:

• Sections

• Entries: Key/Value

• Indented Configuration Mode

The docker events input plugin uses the docker API to capture server events. A complete list of possible events returned by this plugin can be found

Configuration Parameters

This plugin supports the following configuration parameters:

Note: This plugin is experimental and may be unstable. Use it in development or testing environments only, as its features and behavior are subject to change.

The in_ebpf input plugin is an experimental plugin for Fluent Bit that uses eBPF (extended Berkeley Packet Filter) to capture low-level system events. This plugin allows Fluent Bit to monitor kernel-level activities such as process executions, file accesses, memory allocations, network connections, and signal handling. It provides valuable insights into system behavior for debugging, monitoring, and security analysis.

Overview

The in_ebpf

Fluent Bit exposes its to allow you to monitor the internals of your pipeline. The collected metrics can be processed similarly to those from the . They can be sent to output plugins including , or ..

Important note: Metrics collected with Node Exporter Metrics flow through a separate pipeline from logs and current filters do not operate on top of metrics.

Configuration

Fluent Bit supports configuring an HTTP proxy for all egress HTTP/HTTPS traffic using the HTTP_PROXY or http_proxy environment variable.

The format for the HTTP proxy environment variable is http://USER:PASS@HOST:PORT, where:

• USER is the username when using basic authentication.

You may wish to test a logging pipeline locally to observe how it deals with log messages. The following is a walk-through for running Fluent Bit and Elasticsearch locally with which can serve as an example for testing other plugins locally.

Create a Configuration File

Refer to the to create a configuration to test.

fluent-bit.conf:

Fluent Bit, including its core, plugins, and tools are distributed under the terms of the Apache License v2.0:

Container deployment

Install on Linux (packages)

Install on Windows (packages)

Install on macOS (packages)

Compile from Source (Linux, Windows, FreeBSD, macOS)

Sandbox Environment

If you are interested in learning about Fluent Bit you can try out the sandbox environment:

Enterprise Packages

Fluent Bit packages are also provided by for older end of life versions, Unix systems, and additional support and features including aspects like CVE backporting.

Currently, Fluent Bit supports two configuration formats:

• Yaml: standard configuration format as of v3.2.

• Classic mode: to be deprecated at the end of 2025.

Command line interface

Fluent Bit exposes most of it features through the command line interface. Running the -h option you can get a list of the options available:

The pipeline section defines the flow of how data is collected, processed, and sent to its final destination. It encompasses the following core concepts:

Example Configuration

Here’s a simple example of a pipeline configuration:

Pipeline Processors

Processors operate on specific signals such as logs, metrics, and traces. They are attached to an input plugin and must specify the signal type they will process.

Example of a Processor

In the example below, the content_modifier processor inserts or updates (upserts) the key my_new_key with the value 123 for all log records generated by the tail plugin. This processor is only applied to log signals:

Here is a more complete example with multiple processors:

You might noticed that processors not only can be attached to input, but also to an output.

How Are Processors Different from Filters?

While processors and filters are similar in that they can transform, enrich, or drop data from the pipeline, there is a significant difference in how they operate:

• Processors: Run in the same thread as the input plugin when the input plugin is configured to be threaded (threaded: true). This design provides better performance, especially in multi-threaded setups.

• Filters: Run in the main event loop. When multiple filters are used, they can introduce performance overhead, particularly under heavy workloads.

Running Filters as Processors

You can configure existing to run as processors. There are no specific changes needed; you simply use the filter name as if it were a native processor.

Example of a Filter Running as a Processor

In the example below, the grep filter is used as a processor to filter log events based on a pattern:

The elasticsearch input plugin handles both Elasticsearch and OpenSearch Bulk API requests.

Configuration Parameters

The plugin supports the following configuration parameters:

Note: The Elasticsearch cluster uses "sniffing" to optimize the connections between its cluster and clients. Elasticsearch can build its cluster and dynamically generate a connection list which is called "sniffing". The hostname will be used for sniffing information and this is handled by the sniffing endpoint.

Getting Started

In order to start performing the checks, you can run the plugin from the command line or through the configuration file:

Command Line

From the command line you can configure Fluent Bit to handle Bulk API requests with the following options:

Configuration File

In your main configuration file append the following Input & Output sections:

As described above, the plugin will handle ingested Bulk API requests. For large bulk ingestions, you may have to increase buffer size with buffer_max_size and buffer_chunk_size parameters:

Ingesting from beats series

Ingesting from beats series agents is also supported. For example, , , and are able to ingest their collected data through this plugin.

Note that Fluent Bit's node information is returning as Elasticsearch 8.0.0.

So, users have to specify the following configurations on their beats configurations:

For large log ingestion on these beat plugins, users might have to configure rate limiting on those beats plugins when Fluent Bit indicates that the application is exceeding the size limit for HTTP requests:

It's common that Fluent Bit output plugins aims to connect to external services to deliver the logs over the network, this is the case of HTTP, Elasticsearch and Forward within others. Being able to connect to one node (host) is normal and enough for more of the use cases, but there are other scenarios where balancing across different nodes is required. The Upstream feature provides such capability.

An Upstream defines a set of nodes that will be targeted by an output plugin, by the nature of the implementation an output plugin must support the Upstream feature. The following plugin(s) have Upstream support:

• Forward

The current balancing mode implemented is round-robin.

Configuration

To define an Upstream it's required to create an specific configuration file that contains an UPSTREAM and one or multiple NODE sections. The following table describe the properties associated to each section. Note that all of them are mandatory:

Nodes and specific plugin configuration

A Node might contain additional configuration keys required by the plugin, on that way we provide enough flexibility for the output plugin, a common use case is Forward output where if TLS is enabled, it requires a shared key (more details in the example below).

Nodes and TLS (Transport Layer Security)

In addition to the properties defined in the table above, the network operations against a defined node can optionally be done through the use of TLS for further encryption and certificates use.

The TLS options available are described in the section and can be added to the any Node section.

Configuration File Example

The following example defines an Upstream called forward-balancing which aims to be used by Forward output plugin, it register three Nodes:

• node-1: connects to 127.0.0.1:43000

• node-2: connects to 127.0.0.1:44000

• node-3: connects to 127.0.0.1:45000 using TLS without verification. It also defines a specific configuration option required by Forward output called shared_key.

Note that every Upstream definition must exists on it own configuration file in the file system. Adding multiple Upstreams in the same file or different files is not allowed.

Fluent Bit supports the following operating systems and architectures:

From an architecture support perspective, Fluent Bit is fully functional on x86_64, Arm64v8, and Arm32v7 based processors.

Fluent Bit can work also on macOS and Berkeley Software Distribution (BSD) systems, but not all plugins will be available on all platforms.

Official support is based on community demand. Fluent Bit might run on older operating systems, but must be built from source, or using custom packages from .

Fluent Bit is supported for Linux on IBM Z (s390x) environments with some restrictions, but only container images are provided for these targets officially.

The Kafka input plugin allows subscribing to one or more Kafka topics to collect messages from an Apache Kafka service. This plugin uses the official librdkafka C library (built-in dependency).

Configuration Parameters

Getting Started

In order to subscribe/collect messages from Apache Kafka, you can run the plugin from the command line or through the configuration file:

Command Line

The kafka plugin can read parameters through the -p argument (property), e.g:

Configuration File

In your main configuration file append the following Input & Output sections:

Example of using kafka input/output plugins

The Fluent Bit source repository contains a full example of using Fluent Bit to process Kafka records:

The above will connect to the broker listening on kafka-broker:9092 and subscribe to the fb-source topic, polling for new messages every 100 milliseconds.

Since the payload will be in json format, we ask the plugin to automatically parse the payload with format json.

Every message received is then processed with kafka.lua and sent back to the fb-sink topic of the same broker.

The example can be executed locally with make start in the examples/kafka_filter directory (docker/compose is used).

It's possible for logs or data to be ingested or created faster than the ability to flush it to some destinations. A common scenario is when reading from big log files, especially with a large backlog, and dispatching the logs to a backend over the network, which takes time to respond. This generates backpressure, leading to high memory consumption in the service.

To avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts the amount of data an input plugin can ingest. Restriction is done through the configuration parameters Mem_Buf_Limit and storage.Max_Chunks_Up.

As described in the Buffering concepts section, Fluent Bit offers two modes for data handling: in-memory only (default) and in-memory and filesystem (optional).

The default storage.type memory buffer can be restricted with Mem_Buf_Limit. If memory reaches this limit and you reach a backpressure scenario, you won't be able to ingest more data until the data chunks that are in memory can be flushed. The input pauses and Fluent Bit a [warn] [input] {input name or alias} paused (mem buf overlimit) log message.

Depending on the input plugin in use, this might cause incoming data to be discarded (for example, TCP input plugin). The tail plugin can handle pauses without data ingloss, storing its current file offset and resuming reading later. When buffer memory is available, the input resumes accepting logs. Fluent Bit a [info] [input] {input name or alias} resume (mem buf overlimit) message.

Mitigate the risk of data loss by configuring secondary storage on the filesystem using the storage.type of filesystem (as described in ). Initially, logs will be buffered to both memory and the filesystem. When the storage.max_chunks_up limit is reached, all new data will be stored in the filesystem. Fluent Bit stops queueing new data in memory and buffers only to the filesystem. When storage.type filesystem is set, the Mem_Buf_Limit setting no longer has any effect. Instead, the [SERVICE] level storage.max_chunks_up setting controls the size of the memory buffer.

Mem_Buf_Limit

Mem_Buf_Limit applies only with the default storage.type memory. This option is disabled by default and can be applied to all input plugins.

As an example situation:

• Mem_Buf_Limit is set to 1MB.

• The input plugin tries to append 700 KB.

• The engine routes the data to an output plugin.

• The output plugin backend (HTTP Server) is down.

In this situation, the engine allows appending those 500 KB of data into the memory, with a total of 1.2 MB of data buffered. The limit is permissive and will allow a single write past the limit. When the limit is exceeded, the following actions are taken:

• Block local buffers for the input plugin (can't append more data).

• Notify the input plugin, invoking a pause callback.

The engine protects itself and won't append more data coming from the input plugin in question. It's the responsibility of the plugin to keep state and decide what to do in a paused state.

In a few seconds, if the scheduler was able to flush the initial 700 KB of data or it has given up after retrying, that amount of memory is released and the following actions occur:

• Upon data buffer release (700 KB), the internal counters get updated.

• Counters now are set at 500 KB.

• Because 500 KB isless than 1 MB, it checks the input plugin state.

• If the plugin is paused, it invokes a resume callback.

storage.max_chunks_up

The [SERVICE] level storage.max_chunks_up setting controls the size of the memory buffer. When storage.type filesystem is set, the Mem_Buf_Limit setting no longer has an effect.

The setting behaves similar to the Mem_Buf_Limit scenario when the non-default storage.pause_on_chunks_overlimit is enabled.

When (default) storage.pause_on_chunks_overlimit is disabled, the input won't pause when the memory limit is reached. Instead, it switches to buffering logs only in the filesystem. Limit the disk spaced used for filesystem buffering with storage.total_limit_size.

See docs for more information.

About pause and resume callbacks

Each plugin is independent and not all of them implement pause and resume callbacks. These callbacks are a notification mechanism for the plugin.

One example of a plugin that implements these callbacks and keeps state correctly is the plugin. When the pause callback triggers, it pauses its collectors and stops appending data. Upon resume, it resumes the collectors and continues ingesting data. Tail tracks the current file offset when it pauses, and resumes at the same position. If the file hasn't been deleted or moved, it can still be read.

With the default storage.type memory and Mem_Buf_Limit, the following log messages emit for pause and resume:

With storage.type filesystem and storage.max_chunks_up, the following log messages emit for pause and resume:

Kubernetes exports it events through the API server. This input plugin allows to retrieve those events as logs and get them processed through the pipeline.

Configuration

• * As of Fluent-Bit 3.1, this plugin uses a Kubernetes watch stream instead of polling. In versions before 3.1, the interval parameters are used for reconnecting the Kubernetes watch stream.

Threading

This input always runs in its own .

Getting Started

Kubernetes Service Account

The Kubernetes service account used by Fluent Bit must have get, list, and watch permissions to namespaces and pods for the namespaces watched in the kube_namespace configuration parameter. If you're using the helm chart to configure Fluent Bit, this role is included.

Simple Configuration File

In the following configuration file, the input plugin kubernetes_events collects events every 5 seconds (default for interval_nsec) and exposes them through the on the console.

Event Timestamp

Event timestamps are created from the first existing field, based on the following order of precedence:

1. lastTimestamp

2. firstTimestamp

3. metadata.creationTimestamp

Plugins that interact with AWS services fetch credentials from the following providers in order. Only the first provider that provides credentials is used.

• Environment variables

• Shared configuration and credentials files

• EKS Web Identity Token (OIDC)

All AWS plugins additionally support a role_arn (or AWS_ROLE_ARN, for ) configuration parameter. If specified, the fetched credentials are used to assume the given role.

Environment variables

Plugins use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (and optionally AWS_SESSION_TOKEN) environment variables if set.

Shared configuration and credentials files

Plugins read the shared config file at $AWS_CONFIG_FILE (or $HOME/.aws/config), and the shared credentials file at $AWS_SHARED_CREDENTIALS_FILE (or $HOME/.aws/credentials) to fetch the credentials for the profile named $AWS_PROFILE or $AWS_DEFAULT_PROFILE (or "default"). See .

The shared settings evaluate in the following order:

No other settings are supported.

EKS Web Identity Token (OIDC)

Credentials are fetched using a signed web identity token for a Kubernetes service account. See .

ECS HTTP credentials endpoint

Credentials are fetched for the ECS task's role. See .

EC2 instance profile credentials (IMDS)

Fetches credentials for the EC2 instance profile's role. See . As of Fluent Bit version 1.8.8, IMDSv2 is used by default and IMDSv1 might be disabled. Prior versions of Fluent Bit require enabling IMDSv1 on EC2.

The mem input plugin, gathers the information about the memory and swap usage of the running system every certain interval of time and reports the total amount of memory and the amount of free available.

Getting Started

In order to get memory and swap usage from your system, you can run the plugin from the command line or through the configuration file:

Command Line

Threading

You can enable the threaded setting to run this input in its own .

Configuration File

In your main configuration file append the following Input & Output sections:

The kmsg input plugin reads the Linux Kernel log buffer since the beginning, it gets every record and parse it field as priority, sequence, seconds, useconds, and message.

Configuration Parameters

Getting Started

In order to start getting the Linux Kernel messages, you can run the plugin from the command line or through the configuration file:

Command Line

As described above, the plugin processed all messages that the Linux Kernel reported, the output has been truncated for clarification.

Configuration File

In your main configuration file append the following Input & Output sections:

The MQTT input plugin, allows to retrieve messages/data from MQTT control packets over a TCP connection. The incoming data to receive must be a JSON map.

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

In order to start listening for MQTT messages, you can run the plugin from the command line or through the configuration file:

Command Line

Since the MQTT input plugin let Fluent Bit behave as a server, we need to dispatch some messages using some MQTT client, in the following example mosquitto tool is being used for the purpose:

The following command line will send a message to the MQTT input plugin:

Configuration File

In your main configuration file append the following Input & Output sections:

Fluent Bit works internally with structured records and it can be composed of an unlimited number of keys and values. Values can be anything like a number, string, array, or a map.

Having a way to select a specific part of the record is critical for certain core functionalities or plugins, this feature is called Record Accessor.

consider Record Accessor a simple grammar to specify record content and other miscellaneous values.

Format

A record accessor rule starts with the character $

The service section defines global properties of the service. The available configuration keys are:

has an engine that helps to coordinate the data ingestion from input plugins. The engine calls the scheduler to decide when it's time to flush the data through one or multiple output plugins. The scheduler flushes new data at a fixed number of seconds, and retries when asked.

When an output plugin gets called to flush some data, after processing that data it can notify the engine using these possible return statuses:

• OK: Data successfully processed and flushed.

• Retry

Fluent Bit is distributed as the fluent-bit package and is available for the latest stable CentOS system.

Fluent Bit supports the following architectures:

• x86_64

• aarch64

Fluent Bit is compatible with the latest Apple macOS software for x86_64 and Apple Silicon architectures.

Installation packages

Installation packages can be found .

The cpu input plugin, measures the CPU usage of a process or the whole system by default (considering per CPU core). It reports values in percentage unit for every interval of time set. At the moment this plugin is only available for Linux.

The following tables describes the information generated by the plugin. The keys below represent the data used by the overall system, all values associated to the keys are in a percentage unit (0 to 100%):

The CPU metrics plugin creates metrics that are log-based, such as JSON payload. For Prometheus-based metrics, see the Node Exporter Metrics input plugin.

The exec_wasi input plugin, allows to execute WASM program that is WASI target like as external program and collects event logs from there.

Configuration Parameters

The plugin supports the following configuration parameters:

Health input plugin allows you to check how healthy a TCP server is. It does the check by issuing a TCP connection every a certain interval of time.

Configuration Parameters

The plugin supports the following configuration parameters:

The netif input plugin gathers network traffic information of the running system every certain interval of time, and reports them.

The Network I/O Metrics plugin creates metrics that are log-based, such as JSON payload. For Prometheus-based metrics, see the Node Exporter Metrics input plugin.

Configuration Parameters

The plugin supports the following configuration parameters: