Fluent Bit is distributed as the fluent-bit package and is available for Amazon Linux 2 and Amazon Linux 2023. The following architectures are supported:

• x86_64

• aarch64 / arm64v8

Install on Amazon EC2

To install Fluent Bit and related AWS output plugins on Amazon Linux 2 on EC2 using AWS Systems Manager, follow this AWS guide.

General installation

To install Fluent Bit on any Amazon Linux instance, follow these steps.

1. Fluent Bit is provided through a Yum repository. To add the repository reference to your system, add a new file called fluent-bit.repo in /etc/yum.repos.d/ with the following content:

[fluent-bit]
  name = Fluent Bit
  baseurl = https://packages.fluentbit.io/amazonlinux/2/
  gpgcheck=1
  gpgkey=https://packages.fluentbit.io/fluentbit.key
  enabled=1

[fluent-bit]
  name = Fluent Bit
  baseurl = https://packages.fluentbit.io/amazonlinux/2023/
  gpgcheck=1
  gpgkey=https://packages.fluentbit.io/fluentbit.key
  enabled=1

1. Ensure your GPG key is up to date.

2. After your repository is configured, run the following command to install it:

3. Instruct systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default Fluent Bit configuration collect metrics of CPU usage and sends the records to the standard output. You can see the outgoing data in your /var/log/messages file.

Fluent Bit is distributed as the fluent-bit package and is available for the latest stable Debian system.

The following architectures are supported

• x86_64

• aarch64

• arm64v8

The recommended secure deployment approach is to use the following instructions:

Server GPG key

The first step is to add the Fluent Bit server GPG key to your keyring to ensure you can get the correct signed packages.

Follow the official Debian wiki guidance.

Update your sources lists

For Debian, you must add the Fluent Bit APT server entry to your sources lists. Ensure codename is set to your specific Debian release name. (for example: bookworm for Debian 12).

Update your source's lists:

Update your repositories database

Update your system's apt database:

Install Fluent Bit

1. Ensure your GPG key is up to date.

2. Use the following apt-get command to install the latest Fluent Bit:

3. Instruct systemd to enable the service:

If you do a status check, you should see a similar output similar to:

The default Fluent Bit configuration collect metrics of CPU usage and sends the records to the standard output. You can see the outgoing data in your /var/log/messages file.

Fluent Bit is distributed as the fluent-bit package and is available for Raspberry Pi. The following versions are supported:

• Raspbian Bookworm (12)

• Raspbian Bullseye (11)

• Raspbian Buster (10)

Server GPG key

The first step is to add the Fluent Bit server GPG key to your keyring so you can get Fluent Bit signed packages:

Update your sources lists

On Debian and derivative systems such as Raspbian, you need to add the Fluent Bit APT server entry to your sources lists.

Add the following content at bottom of your /etc/apt/sources.list file.

Raspbian 12 (Bookworm)

Raspbian 11 (Bullseye)

Raspbian 10 (Buster)

Update your repositories database

Now let your system update the apt database:

Install Fluent Bit

1. Ensure your GPG key is up to date.

2. Use the following apt-get command to install the latest Fluent Bit:

3. Instruct systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default configuration of Fluent Bit collects metrics for CPU usage and sends the records to the standard output. You can see the outgoing data in your /var/log/syslog file.

Fluent Bit is distributed as the fluent-bit package and is available for the latest versions of Rocky or Alma Linux now that CentOS Stream is tracking more recent dependencies.

Fluent Bit supports the following architectures:

• x86_64

• aarch64

• arm64v8

RHEL 9

From CentOS 9 Stream and later, the CentOS dependencies will update more often than downstream usage. This might mean that incompatible (more recent) versions are provided of certain dependencies (for example, OpenSSL). For OSS, there are RockyLinux and AlmaLinux repositories. This might be required for RHEL 9 as well which will no longer track equivalent CentOS 9 stream dependencies. No RHEL 9 build is provided, it is expected to use one of the OSS variants listed.

Configure YUM

The fluent-bit package is provided through a Yum repository. To add the repository reference to your system:

1. In /etc/yum.repos.d/, add a new file called fluent-bit.repo.

2. Add the following content to the file - replace almalinux with rockylinux if required:

3. As a best practice, enable gpgcheck and repo_gpgcheck for security reasons. Fluent Bit signs its repository metadata and all Fluent Bit packages.

Install

1. After your repository is configured, run the following command to install it:

2. Instruct Systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default Fluent Bit configuration collect metrics of CPU usage and sends the records to the standard output. You can see the outgoing data in your /var/log/messages file.

Fluent Bit is distributed as the fluent-bit package and is available for long-term support releases of Ubuntu. The latest officially supported version is Ubuntu 26.04. Long-term support releases are also available for Noble Numbat (24.04) and Jammy Jellyfish (22.04).

The recommended secure deployment approach is to use the following instructions.

Server GPG key

Add the Fluent Bit server GPG key to your keyring to ensure you can get the correct signed packages.

Follow the official Debian wiki guidance.

Update your sources lists

On Ubuntu, you need to add the Fluent Bit APT server entry to your sources lists. Ensure codename is set to your specific Ubuntu release name. For example, focal for Ubuntu 20.04.

Update your source's list:

Update your repositories database

Update the apt database on your system:

Install Fluent Bit

1. Ensure your GPG key is up to date.

2. Use the following apt-get command to install the latest Fluent Bit:

3. Instruct systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output. You can see the outgoing data in your /var/log/syslog file.

Fluent Bit is compatible with the latest Apple macOS software for x86_64 and Apple Silicon architectures.

Installation packages

Installation packages can be found in the Fluent Bit repository.

Requirements

You must have Homebrew installed in your system. If it isn't present, install it with the following command:

Installing from Homebrew

The Fluent Bit package on Homebrew isn't officially supported, but should work for basic use cases and testing. It can be installed using:

Compile from source

Install build dependencies

Run the following brew command in your terminal to retrieve the dependencies:

Download and build the source

1. Download a copy of the Fluent Bit source code (upstream):

1. Go to the Fluent Bit directory.

If you want to use a specific version, checkout to the proper tag. For example, to use v4.0.4, use the command:

1. To prepare the build system, you must export certain environment variables so Fluent Bit CMake build rules can pick the right libraries:

1. Change to the build/ directory inside the Fluent Bit sources:

1. Build Fluent Bit. This example indicates to the build system the location the final binaries and config files should be installed:

1. Install Fluent Bit to the previously specified directory. Writing to this directory requires root privileges.

The binaries and configuration examples can be located at /opt/fluent-bit/.

Create macOS installer from source

1. Clone the Fluent Bit source code (upstream):

1. Change to the Fluent Bit directory

To use a specific version, checkout to the proper tag. For example, to use v4.0.4 do:

1. To prepare the build system, you must expose certain environment variables so Fluent Bit CMake build rules can pick the right libraries:

1. Create the specific macOS SDK target. For example, to specify macOS Big Sur (11.3) SDK environment:

1. Change to the build/ directory inside the Fluent Bit sources:

1. Build the Fluent Bit macOS installer:

The macOS installer will be generated as:

Finally, the fluent-bit-<fluent-bit version>-(intel or apple).pkg will be generated.

The created installer will put binaries at /opt/fluent-bit/.

Running Fluent Bit

To make the access path easier to Fluent Bit binary, extend the PATH variable:

To test, try Fluent Bit by generating a test message using the Dummy input plugin which prints to the standard output interface every one second:

You will see an output similar to this:

To halt the process, press ctrl-c in the terminal.

The includes section of YAML configuration files lets you specify additional YAML files to be merged into the current configuration. This lets you organize complex configurations into smaller, manageable files and include them as needed.

These files are identified as a list of filenames and can include relative or absolute paths. If a path isn't specified as absolute, it will be treated as relative to the file that includes it.

Usage

The following example demonstrates how to include additional YAML files using relative path references. This is the file system path structure:

├── fluent-bit.yaml
├── inclusion-1.yaml
└── subdir
    └── inclusion-2.yaml

You can reference these files in fluent-bit.yaml as follows:

includes:
  - inclusion-1.yaml
  - subdir/inclusion-2.yaml

Ensure that the included files are formatted correctly and contain valid YAML configurations for seamless integration.

You can define customer parsers in the parsers section of YAML configuration files.

Syntax

To define custom parsers in the parsers section of a YAML configuration file, use the following syntax.

parsers:
  - name: custom_parser1
    format: json
    time_key: time
    time_format: '%Y-%m-%dT%H:%M:%S.%L'
    time_keep: on

  - name: custom_parser2
    format: regex
    regex: '^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$'
    time_key: time
    time_format: '%Y-%m-%dT%H:%M:%S.%L'
    time_keep: on
    types: pid:integer

For information about supported configuration options for custom parsers, see configuring custom parsers.

Standalone parsers files

In addition to defining parsers in the parsers section of YAML configuration files, you can store parser definitions in standalone files. These standalone files require the same syntax as parsers defined in a standard YAML configuration file.

To add a standalone parsers file to Fluent Bit, use the parsers_file parameter in the service section of your YAML configuration file.

Add a standalone parsers file to Fluent Bit

To add a standalone parsers file to Fluent Bit, follow these steps.

1. Define custom parsers in a standalone YAML file. For example, custom-parsers.yaml defines two custom parsers:

1. Update the parsers_file parameter in the service section of your YAML configuration file:

In addition to the plugins that come bundled with Fluent Bit, you can load external plugins. Use this feature for loading Go or WebAssembly (Wasm) plugins that are built as shared object files (.so).

Inline YAML

You can specify external plugins in the plugins section of YAML configuration files. For example:

YAML plugins file included using the plugins_file option

Additionally, you can define external plugins in a separate YAML file, then reference that file in the plugins_file key nested under the service section of your YAML configuration file. For example:

In this setup, the extra_plugins.yaml file might contain the following plugins section:

plugins:
  - /other/path/to/out_gstdout.so

The upstream_servers section of YAML configuration files defines a group of endpoints, referred to as nodes. Nodes are used by output plugins to distribute data in a round-robin fashion. Use this section for plugins that require load balancing when sending data. Examples of plugins that support this capability include Forward and Elasticsearch.

The upstream_servers section require specifying a name for the group and a list of nodes. The following example defines two upstream server groups, forward-balancing and forward-balancing-2:

upstream_servers:
  - name: forward-balancing
    nodes:
      - name: node-1
        host: 127.0.0.1
        port: 43000

      - name: node-2
        host: 127.0.0.1
        port: 44000

      - name: node-3
        host: 127.0.0.1
        port: 45000
        tls: true
        tls_verify: false
        shared_key: secret

  - name: forward-balancing-2
    nodes:
      - name: node-A
        host: 192.168.1.10
        port: 50000

      - name: node-B
        host: 192.168.1.11
        port: 51000

Each node in the upstream_servers group must specify a name, host, and port. Additional settings like tls, tls_verify, and shared_key can be configured for secure communication.

While the upstream_servers section can be defined globally, some output plugins might require the configuration to be specified in a separate YAML file. Consult the documentation for each specific output plugin to understand its requirements.

Classic mode is a custom configuration model for Fluent Bit. It's more limited than the YAML configuration mode, and doesn't have the more extensive feature support the YAML configuration has. Classic mode basic design only supports grouping sections with key-value pairs and lacks the ability to handle sub-sections or complex data structures like lists.

Learn more about classic mode:

• Format and schema

• Variables

• Configuration file

• Commands

• Upstream servers

• Record accessor syntax

Fluent Bit might optionally use a configuration file to define how the service will behave.

The schema is defined by three concepts:

• Sections

• Entries: key/value

• Indented Configuration Mode

An example of a configuration file is as follows:

Sections

A section is defined by a name or title inside brackets. Using the previous example, a Service section has been set using [SERVICE] definition. The following rules apply:

• All section content must be indented (four spaces ideally).

• Multiple sections can exist on the same file.

• A section must have comments and entries.

• Any commented line under a section must be indented too.

• End-of-line comments aren't supported, only full-line comments.

Entries: key/value

A section can contain entries. An entry is defined by a line of text that contains a Key and a Value. Using the previous example, the [SERVICE] section contains two entries: one is the key Daemon with value off and the other is the key Log_Level with the value debug. The following rules apply:

• An entry is defined by a key and a value.

• A key must be indented.

• A key must contain a value which ends in a line break.

• Multiple keys with the same name can exist.

Commented lines are set prefixing the # character. Commented lines aren't processed but they must be indented.

Indented configuration mode

Fluent Bit configuration files are based in a strict indented mode. Each configuration file must follow the same pattern of alignment from left to right when writing text. By default, an indentation level of four spaces from left to right is suggested. Example:

This example shows two sections with multiple entries and comments. Empty lines are allowed.

Fluent Bit works internally with structured records and it can be composed of an unlimited number of keys and values. Values can be anything like a number, string, array, or a map.

Having a way to select a specific part of the record is critical for certain core functionalities or plugins, this feature is called Record Accessor.

Consider record accessor to be a basic grammar to specify record content and other miscellaneous values.

Format

A record accessor rule starts with the character $. Use the structured content as an example. The following table describes how to access a record:

The following table describes some accessing rules and the expected returned value:

If the accessor key doesn't exist in the record like the last example $labels['undefined'], the operation is omitted, and no exception will occur.

Usage

The feature is enabled on a per plugin basis. Not all plugins enable this feature. As an example, consider a configuration that aims to filter records using grep that only matches where labels have a color blue:

The file content to process in test.log is the following:

When running Fluent Bit with the previous configuration, the output is:

Limitations of record_accessor templating

The Fluent Bit record_accessor library has a limitation in the characters that can separate template variables. Only dots and commas (. and ,) can come after a template variable. This is because the templating library must parse the template and determine the end of a variable.

The following templates are invalid because the template variables aren't separated by commas or dots:

• $TaskID-$ECSContainerName

• $TaskID/$ECSContainerName

• $TaskID_$ECSContainerName

• $TaskIDfooo$ECSContainerName

However, the following are valid:

• $TaskID.$ECSContainerName

• $TaskID.ecs_resource.$ECSContainerName

• $TaskID.fooo.$ECSContainerName

And the following are valid since they only contain one template variable with nothing after it:

• fooo$TaskID

• fooo____$TaskID

• fooo/bar$TaskID

Fluent Bit supports the reloading feature when enabled in the configuration file or on the command line with -Y or --enable-hot-reload option.

Hot reloading is supported on Linux, macOS, and Windows operating systems.

Update the configuration

To get started with reloading over HTTP, enable the HTTP Server in the configuration file:

service:
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

[SERVICE]
  HTTP_Server  On
  HTTP_Listen  0.0.0.0
  HTTP_PORT    2020
  Hot_Reload   On

How to reload

After updating the configuration, use one of the following methods to perform a hot reload:

HTTP

Use the following HTTP endpoints to perform a hot reload:

• PUT /api/v2/reload

• POST /api/v2/reload

For using curl to reload Fluent Bit, users must specify an empty request body as:

Obtain a count of hot reload using the HTTP endpoint:

• GET /api/v2/reload

The endpoint returns hot_reload_count as follows:

The default value of the counter is 0.

Signal

Hot reloading can be used with SIGHUP.

SIGHUP signal isn't supported on Windows.

Confirm a reload

Use one of the following methods to confirm the reload occurred.

Fluent Bit has one event loop to handle critical operations, like managing timers, receiving internal messages, scheduling flushes, and handling retries. This event loop runs in the main Fluent Bit thread.

To free up resources in the main thread, you can configure inputs and outputs to run in their own self-contained threads. However, inputs and outputs implement multithreading in distinct ways: inputs can run in threaded mode, and outputs can use one or more workers.

Threading also affects certain processes related to inputs and outputs. For example, filters always run in the main thread, but processors run in the self-contained threads of their respective inputs or outputs, if applicable.

Inputs

When inputs collect telemetry data, they can either perform this process inside the main Fluent Bit thread or inside a separate dedicated thread. You can configure this behavior by enabling or disabling the threaded setting.

All inputs are capable of running in threaded mode, but certain inputs always run in threaded mode regardless of configuration. These always-threaded inputs are:

• Kubernetes Events

• Node Exporter Metrics

• Process Exporter Metrics

• Windows Exporter Metrics

Inputs aren't internally aware of multithreading. If an input runs in threaded mode, Fluent Bit manages the logistics of that input's thread.

Outputs

When outputs flush data, they can either perform this operation inside the main Fluent Bit thread or inside a separate dedicated thread called a worker. Each output can have one or more workers running in parallel, and each worker can handle multiple concurrent flushes. You can configure this behavior by changing the value of the workers setting.

All outputs are capable of running in multiple workers, and each output has a default value of 0, 1, or 2 workers. However, even if an output uses workers by default, you can safely reduce the number of workers under the default or disable workers entirely.

Fluent Bit is an open source telemetry agent that processes logs, metrics, traces, and profiles. It's designed to efficiently handle the challenges of collecting and processing telemetry data across a wide range of environments, from constrained systems to complex cloud infrastructures. Managing telemetry data from various sources and formats can be a constant challenge, particularly when performance is a critical factor.

Rather than serving as a drop-in replacement, Fluent Bit enhances the observability strategy for your infrastructure. It adapts and optimizes your existing logging layer, and adds metrics and traces processing. Fluent Bit supports a vendor-neutral approach, with native OpenTelemetry (OTLP) ingestion and delivery and seamless integration with ecosystems such as Prometheus. Trusted by major cloud providers, banks, and companies that need a ready-to-use telemetry agent, Fluent Bit effectively manages diverse data sources and formats. It maintains optimal performance while keeping resource consumption low.

Fluent Bit can be deployed as an edge agent for localized telemetry data handling or utilized as a central aggregator/collector for managing telemetry data across multiple sources and environments.

The history of Fluent Bit

In 2014, the Fluentd team at Treasure Data was forecasting the need for a lightweight log processor for constraint environments like embedded Linux and gateways. To meet this need, Eduardo Silva created Fluent Bit, a new open source solution and part of the Fluentd ecosystem.

After the project matured, it gained traction for normal Linux systems. With the new containerized world, the cloud native community asked to extend the project scope to support more sources, filters, and destinations. Not long after, Fluent Bit became one of the preferred solutions to solve the logging challenges in cloud environments.

Telemetry data processing can be complex, especially at scale. That's why Fluentd was created. Fluentd is more than a basic tool. It's grown into a full-scale ecosystem that contains SDKs for different languages and sub-projects, like Fluent Bit.

The Fluentd and Fluent Bit projects are both:

• Licensed under the terms of Apache License v2.0.

• Graduated hosted projects by the Cloud Native Computing Foundation (CNCF).

• Production-grade solutions, with Fluent Bit deployed over 15 billion times globally.

• Vendor neutral and community driven.

• Widely adopted by the industry, being trusted by major companies like Amazon, Microsoft, Google, and hundreds of others.

The projects have many similarities: Fluent Bit is designed and built on top of the best ideas of Fluentd architecture and general design. Which one you choose depends on your end-users' needs.

The following table describes a comparison of different areas of the projects:

Both Fluentd and Fluent Bit can work as Aggregators or Forwarders, and can complement each other or be used as standalone solutions.

In the recent years, cloud providers have switched from Fluentd to Fluent Bit for performance and compatibility. Fluent Bit is now considered the next-generation solution.

To use these lab resources to their full extent, you'll need to install resources in your local environment to test and run Fluent Bit.

O11y workshops by Chronosphere

Chronosphere provides several open source observability workshops, including a Fluent Bit workshop:

1. Lab 1 - Introduction to Fluent Bit

2. Lab 2 - Installing Fluent Bit

3. Lab 3 - Exploring First Pipelines

4. Lab 4 - Exploring More Pipelines

5. Lab 5 - Understanding Backpressure

6. Lab 6 - Avoid Telemetry Data Loss

7. Lab 7 - Pipeline Integration with OpenTelemetry

8. Lab 8 - Controlling logs with Fluent Bit on Kubernetes

9. Lab 9 - Visualizing Fluent Bit telemetry data with OpenSearch

10. Lab 10 - Advanced telemetry routing

You can also view the source files for these workshops on GitLab.

Fluent Bit pod logging on Kubernetes workshop by Amazon

This workshop by Amazon covers deploying Fluent Bit for pod-level logging on Kubernetes and routing data to CloudWatch Logs.

Before diving into Fluent Bit you might want to get acquainted with some of the key concepts of the service. This document provides an introduction to those concepts and common Fluent Bit terminology. Reading this document will help you gain a more general understanding of the following topics:

• Event or Record

• Filtering

• Processor

• Tag

• Timestamp

• Match

• Structured Message

Events or records

Every incoming piece of data that belongs to a log, metric, trace, or profile that's retrieved by Fluent Bit is considered an Event or a Record.

As an example, consider the following content of a Syslog file:

Jan 18 12:52:16 flb systemd[2222]: Starting GNOME Terminal Server
Jan 18 12:52:16 flb dbus-daemon[2243]: [session uid=1000 pid=2243] Successfully activated service 'org.gnome.Terminal'
Jan 18 12:52:16 flb systemd[2222]: Started GNOME Terminal Server.
Jan 18 12:52:16 flb gsd-media-keys[2640]: # watch_fast: "/org/gnome/terminal/legacy/" (establishing: 0, active: 0)

It contains four lines that represent four independent Events.

An Event is comprised of:

• timestamp

• key/value metadata (v2.1.0 and greater)

• payload

Event format

The Fluent Bit wire protocol represents an Event as a two-element array with a nested array as the first element:

[[TIMESTAMP, METADATA], MESSAGE]

where

• TIMESTAMP is a timestamp in seconds as an integer or floating point value (not a string).

• METADATA is an object containing event metadata, and might be empty.

• MESSAGE is an object containing the event body.

Fluent Bit versions prior to v2.1.0 used:

[TIMESTAMP, MESSAGE]

to represent events. This format is still supported for reading input event streams.

Filtering

You might need to perform modifications on an event's content. The process to alter, append to, or drop Events is called filtering.

Use filtering to:

• Append specific information to the Event like an IP address or metadata.

• Select a specific piece of the Event content.

• Drop Events that match a certain pattern.

Processor

Processors modify, transform, or enhance data as it moves through the pipeline. Unlike filters, processors are attached directly to individual input or output plugins and don't use tag matching. Each processor operates only on data from its associated plugin.

Processors run in the same thread as their associated plugin, which improves throughput compared to filters—especially when multithreading is enabled.

Processors are supported in YAML configuration files only, and can act on logs, metrics, traces, and profiles.

Tag

Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string used in a later stage by the Router to decide which Filter or Output phase it must go through.

Most tags are assigned manually in the configuration. If a tag isn't specified, Fluent Bit assigns the name of the Input plugin instance where that Event was generated from.

A tagged record must always have a Matching rule. To learn more about Tags and Matches, see Routing.

Timestamp

The timestamp represents the time an Event was created. Every Event contains an associated timestamp. All events have timestamps, and they're set by the input plugin or discovered through a data parsing process.

The timestamp is a numeric fractional integer in the format:

SECONDS.NANOSECONDS

where:

• _SECONDS_ is the number of seconds that have elapsed since the Unix epoch.

• _NANOSECONDS_ is a fractional second or one thousand-millionth of a second.

Match

Fluent Bit lets you route your collected and processed Events to one or multiple destinations. A Match represents a rule to select Events where a Tag matches a defined rule.

To learn more about Tags and Matches, see Routing.

Structured messages

Source events can have a structure. A structure defines a set of keys and values inside the Event message to implement faster operations on data modifications. Fluent Bit treats every Event message as a structured message.

Consider the following two messages:

• No structured message

  Copy

  "Project Fluent Bit created on 1398289291"

• With a structured message

  Copy

  {"project": "Fluent Bit", "created": 1398289291}

For performance reasons, Fluent Bit uses a binary serialization data format called MessagePack.

Fluent Bit uses CMake as its build system.

Requirements

To build and install Fluent Bit from source, you must also install the following packages:

• bison

• build-essentials

• cmake (version 3.31.6 or later)

• flex

• libssl-dev

• libyaml-dev

• pkg-config

Additionally, certain input or output plugins might depend on additional components. For example, some plugins require Kafka.

Prepare environment

If you already know how CMake works, you can skip this section and review the available build options.

The following steps explain how to build and install the project with the default options.

1. Change to the build/ directory inside the Fluent Bit sources:

   Copy

   cd build/

2. Let CMake configure the project specifying where the root path is located:

   Copy

   cmake ../

   This command displays a series of results similar to:

   Copy

   -- The C compiler identification is GNU 4.9.2
   -- Check for working C compiler: /usr/bin/cc
   -- Check for working C compiler: /usr/bin/cc -- works
   -- Detecting C compiler ABI info
   -- Detecting C compiler ABI info - done
   -- The CXX compiler identification is GNU 4.9.2
   -- Check for working CXX compiler: /usr/bin/c++
   -- Check for working CXX compiler: /usr/bin/c++ -- works
   ...
   -- Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE)
   -- Looking for accept4
   -- Looking for accept4 - not found
   -- Configuring done
   -- Generating done
   -- Build files have been written to: /home/edsiper/coding/fluent-bit/build

3. Start the compilation process using the make command:

   This command displays results similar to:

4. To continue installing the binary on the system, use make install:

   If the command indicates insufficient permissions, prefix the command with sudo.

Build options

Fluent Bit provides configurable options to CMake that can be enabled or disabled.

General options

Development options

Optimization options

Input plugins

Input plugins gather information from a specific source type like network interfaces, some built-in metrics, or through a specific input device.

The following input plugins are available:

Processor plugins

Processor plugins handle the events within the processor pipelines to allow modifying, enriching, or dropping events.

The following table describes the processors available:

Filter plugins

Filter plugins let you modify, enrich or drop records.

The following table describes the filters available on this version:

Output plugins

Output plugins let you flush the information to some external interface, service, or terminal.

The following table describes the output plugins available:

Fluent Bit in normal operation mode is configurable through text files or using specific arguments in the command line. Although this is the ideal deployment case, there are scenarios where a more restricted configuration is required. Static configuration mode restricts configuration ability.

Static configuration mode includes a built-in configuration in the final binary of Fluent Bit, disabling the usage of external files or flags at runtime.

Get started

Requirements

The following steps assume you are familiar with configuring Fluent Bit using text files and you have experience building it from scratch as described in Build and Install.

Configuration directory

In your file system, prepare a specific directory that will be used as an entry point for the build system to lookup and parse the configuration files. This directory must contain a minimum of one configuration file, called fluent-bit.conf, that contains the required SERVICE, INPUT, and OUTPUT sections.

As an example, create a new fluent-bit.yaml file or fluent-bit.conf file:

service:
  flush: 1
  daemon

[SERVICE]
  Flush     1
  Daemon    off
  Log_Level info

[INPUT]
  Name      cpu

[OUTPUT]
  Name      stdout
  Match     *

This configuration calculates CPU metrics from the running system and prints them to the standard output interface.

Build with custom configuration

1. Go to the Fluent Bit source code build directory:

   Copy

   cd fluent-bit/build/

2. Run CMake, appending the FLB_STATIC_CONF option pointing to the configuration directory recently created:

   Copy

   cmake -DFLB_STATIC_CONF=/path/to/my/confdir/

3. Build Fluent Bit:

   Copy

   make

The generated fluent-bit binary is ready to run without additional configuration:

$ bin/fluent-bit

...
[0] cpu.local: [1539984752.000347547, {"cpu_p"=>0.750000, "user_p"=>0.500000, "system_p"=>0.250000, "cpu0.p_cpu"=>1.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>0.000000, "cpu1.p_cpu"=>0.000000, "cpu1.p_user"=>0.000000, "cpu1.p_system"=>0.000000, "cpu2.p_cpu"=>0.000000, "cpu2.p_user"=>0.000000, "cpu2.p_system"=>0.000000, "cpu3.p_cpu"=>1.000000, "cpu3.p_user"=>1.000000, "cpu3.p_system"=>0.000000}]

Fluent Bit source code provides BitBake recipes to configure, build, and package the software for a Yocto-based image. Specific steps in the usage of these recipes in your Yocto environment (Poky) is out of the scope of this documentation.

Fluent Bit distributes two main recipes, one for testing/dev purposes and one with the latest stable release.

It's strongly recommended to always use the stable release of the Fluent Bit recipe and not the one from Git master for production deployments.

Fluent Bit and other architectures

Fluent Bit 1.1.x and later fully supports x86_64, x86, arm32v7, and arm64v8.

Fluent Bit is a lightweight and extensible log processor with full support for Kubernetes:

• Process Kubernetes containers logs from the file system or Systemd/Journald.

• Enrich logs with Kubernetes Metadata.

• Centralize your logs in third party storage services like Elasticsearch, InfluxDB, HTTP, and so on.

Concepts

Before getting started it's important to understand how Fluent Bit will be deployed. Kubernetes manages a cluster of nodes. The Fluent Bit log agent tool needs to run on every node to collect logs from every pod. Fluent Bit is deployed as a DaemonSet, which is a pod that runs on every node of the cluster.

When Fluent Bit runs, it reads, parses, and filters the logs of every pod. In addition, Fluent Bit adds metadata to each entry using the Kubernetes filter plugin.

The Kubernetes filter plugin talks to the Kubernetes API Server to retrieve relevant information such as the pod_id, labels, and annotations. Other fields, such as pod_name, container_id, and container_name, are retrieved locally from the log file names. All of this is handled automatically, and no intervention is required from a configuration aspect.

Installation

Fluent Bit should be deployed as a DaemonSet, so it will be available on every node of your Kubernetes cluster.

The recommended way to deploy Fluent Bit for Kubernetes is with the official Helm Chart.

OpenShift

If you are using Red Hat OpenShift you must set up Security Context Constraints (SCC) using the relevant option in the helm chart.

Installing with Helm chart

Helm is a package manager for Kubernetes and lets you deploy application packages into your running cluster. Fluent Bit is distributed using a Helm chart found in the Fluent Helm Charts repository.

Use the following command to add the Fluent Helm charts repository

helm repo add fluent https://fluent.github.io/helm-charts

To validate that the repository was added, run helm search repo fluent to ensure the charts were added. The default chart can then be installed by running the following command:

helm upgrade --install fluent-bit fluent/fluent-bit

Default values

The default chart values include configuration to read container logs. With Docker parsing, Systemd logs apply Kubernetes metadata enrichment, and output to an Elasticsearch cluster. You can modify the included values file to specify additional outputs, health checks, monitoring endpoints, or other configuration options.

Details

The default configuration of Fluent Bit ensures the following:

• Consume all containers logs from the running node and parse them with either the docker or cri multi-line parser.

• Persist how far it got into each file it's tailing so if a pod is restarted it picks up from where it left off.

• The Kubernetes filter adds Kubernetes metadata, specifically labels and annotations. The filter only contacts the API Server when it can't find the cached information, otherwise it uses the cache.

• The default backend in the configuration is Elasticsearch set by the . It uses the Logstash format to ingest the logs. If you need a different Index and Type, refer to the plugin option and update as needed.

• There is an option called Retry_Limit, which is set to False. If Fluent Bit can't flush the records to Elasticsearch, it will retry indefinitely until it succeeds.

Windows deployment

Fluent Bit v1.5.0 and later supports deployment to Windows pods.

Log files overview

When deploying Fluent Bit to Kubernetes, there are three log files that you need to pay attention to.

• C:\k\kubelet.err.log

  This is the error log file from kubelet daemon running on host. Retain this file for future troubleshooting, including debugging deployment failures.

• C:\var\log\containers\<pod>_<namespace>_<container>-<docker>.log

  This is the main log file you need to watch. Configure Fluent Bit to follow this file. It's a symlink to the Docker log file in C:\ProgramData\, with some additional metadata on the file's name.

• C:\ProgramData\Docker\containers\<docker>\<docker>.log

  This is the log file produced by Docker. Normally you don't directly read from this file, but you need to make sure that this file is visible from Fluent Bit.

Typically, your deployment YAML contains the following volume configuration.

spec:
  containers:
  - name: fluent-bit
    image: my-repo/fluent-bit:1.8.4
    volumeMounts:
    - mountPath: C:\k
      name: k
    - mountPath: C:\var\log
      name: varlog
    - mountPath: C:\ProgramData
      name: progdata
  volumes:
  - name: k
    hostPath:
      path: C:\k
  - name: varlog
    hostPath:
      path: C:\var\log
  - name: progdata
    hostPath:
      path: C:\ProgramData

Configure Fluent Bit

Assuming the basic volume configuration described previously, you can apply one of the following configurations to start logging:

parsers:
    - name: docker
      format: json
      time_key: time
      time_format: '%Y-%m-%dT%H:%M:%S.%L'
      time_keep: true

pipeline:
    inputs:
        - name: tail
          tag: kube.*
          path: 'C:\\var\\log\\containers\\*.log'
          parser: docker
          db: 'C:\\fluent-bit\\tail_docker.db'
          mem_buf_limit: 7MB
          refresh_interval: 10

        - name: tail
          tag: kube.error
          path: 'C:\\k\\kubelet.err.log'
          db: 'C:\\fluent-bit\\tail_kubelet.db'

    filters:
        - name: kubernetes
          match: kube.*
          kube_url: 'https://kubernetes.default.svc.cluster.local:443'

    outputs:
        - name: stdout
          match: '*'

fluent-bit.conf: |
    [SERVICE]
      Parsers_File      C:\\fluent-bit\\parsers.conf

    [INPUT]
      Name              tail
      Tag               kube.*
      Path              C:\\var\\log\\containers\\*.log
      Parser            docker
      DB                C:\\fluent-bit\\tail_docker.db
      Mem_Buf_Limit     7MB
      Refresh_Interval  10

    [INPUT]
      Name              tail
      Tag               kubelet.err
      Path              C:\\k\\kubelet.err.log
      DB                C:\\fluent-bit\\tail_kubelet.db

    [FILTER]
      Name              kubernetes
      Match             kube.*
      Kube_URL          https://kubernetes.default.svc.cluster.local:443

    [OUTPUT]
      Name  stdout
      Match *

parsers.conf: |
    [PARSER]
        Name         docker
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On

Mitigate unstable network on Windows pods

Windows pods often lack working DNS immediately after boot (#78479). To mitigate this issue, filter_kubernetes provides a built-in mechanism to wait until the network starts up:

• DNS_Retries: Retries N times until the network start working (6)

• DNS_Wait_Time: Lookup interval between network status checks (30)

By default, Fluent Bit waits for three minutes (30 seconds x 6 times). If it's not enough for you, update the configuration as follows:

    filters:
        - name: kubernetes
          ...
          dns_retries: 10
          dns_wait_time: 30

[filter]
    Name kubernetes
    ...
    DNS_Retries 10
    DNS_Wait_Time 30

Fluent Bit is distributed as the fluent-bit package for Windows and as a Windows container on Docker Hub. Fluent Bit provides two Windows installers: a ZIP archive and an EXE installer.

Not all plugins are supported on Windows. The CMake configuration shows the default set of supported plugins.

Configuration

Provide a valid Windows configuration with the installation.

The following configuration is an example:

service:
  flush: 5
  daemon: off
  log_level: info
  parsers_file: parsers.yaml
  plugins_file: plugins.yaml
  http_server: off
  http_listen: 0.0.0.0
  http_port: 2020
  storage.metrics: on

pipeline:
  inputs:
    - name: winlog
      channels: Setup,Windows Powershell
      interval_sec: 1

  outputs:
    - name: stdout
      match: '*'

[SERVICE]
  # Flush
  # =====
  # set an interval of seconds before to flush records to a destination
  flush        5

  # Daemon
  # ======
  # instruct Fluent Bit to run in foreground or background mode.
  daemon       Off

  # Log_Level
  # =========
  # Set the verbosity level of the service, values can be:
  #
  # - error
  # - warning
  # - info
  # - debug
  # - trace
  #
  # by default 'info' is set, that means it includes 'error' and 'warning'.
  log_level    info

  # Parsers File
  # ============
  # specify an optional 'Parsers' configuration file
  parsers_file parsers.conf

  # Plugins File
  # ============
  # specify an optional 'Plugins' configuration file to load external plugins.
  plugins_file plugins.conf

  # HTTP Server
  # ===========
  # Enable/Disable the built-in HTTP Server for metrics
  http_server  Off
  http_listen  0.0.0.0
  http_port    2020

  # Storage
  # =======
  # Fluent Bit can use memory and filesystem buffering based mechanisms
  #
  # - https://docs.fluentbit.io/manual/administration/buffering-and-storage
  #
  # storage metrics
  # ---------------
  # publish storage pipeline metrics in '/api/v1/storage'. The metrics are
  # exported only if the 'http_server' option is enabled.
  #
  storage.metrics on

[INPUT]
  Name         winlog
  Channels     Setup,Windows PowerShell
  Interval_Sec 1

[OUTPUT]
  name  stdout
  match *

Migration to Fluent Bit

For version 1.9 and later, td-agent-bit is a deprecated package and was removed after 1.9.9. The correct package name to use now is fluent-bit.

Installation packages

The latest stable version is 5.0.7.

These are now using the Github Actions built versions. Legacy AppVeyor builds are still available (AMD 32/64 only) at releases.fluentbit.io but are deprecated.

MSI installers are also available:

• fluent-bit-5.0.7-win32.msi

• fluent-bit-5.0.7-win64.msi

• fluent-bit-5.0.7-winarm64.msi

To check the integrity, use the Get-FileHash cmdlet for PowerShell.

Get-FileHash fluent-bit-5.0.7-win32.exe

Installing from a ZIP archive

1. Download a ZIP archive. Choose the suitable installers for your 32-bit or 64-bit environments.

2. Expand the ZIP archive. You can do this by clicking Extract All in Explorer or Expand-Archive in PowerShell.

   Copy

   Expand-Archive fluent-bit-5.0.7-win64.zip

   The ZIP package contains the following set of files.

   Copy

   fluent-bit
   ├── bin
   │   ├── fluent-bit.dll
   │   └── fluent-bit.exe
   │   └── fluent-bit.pdb
   ├── conf
   │   ├── fluent-bit.conf
   │   ├── parsers.conf
   │   └── plugins.conf
   └── include
       │   ├── flb_api.h
       │   ├── ...
       │   └── flb_worker.h
       └── fluent-bit.h

3. Launch cmd.exe or PowerShell on your machine, and execute fluent-bit.exe:

The following output indicates Fluent Bit is running:

fluent-bit.exe  -i dummy -o stdout

...
[0] dummy.0: [1561684385.443823800, {"message"=>"dummy"}]
[1] dummy.0: [1561684386.428399000, {"message"=>"dummy"}]
[2] dummy.0: [1561684387.443641900, {"message"=>"dummy"}]
[3] dummy.0: [1561684388.441405800, {"message"=>"dummy"}]

To halt the process, press Control+C in the terminal.

Installing from the executable installer

1. Download an EXE installer for the appropriate 32-bit or 64-bit build.

2. Double-click the EXE installer you've downloaded. The installation wizard starts.

3. Click Next and finish the installation. By default, Fluent Bit is installed in C:\Program Files\fluent-bit\.

& "C:\Program Files\fluent-bit\bin\fluent-bit.exe" -i dummy -o stdout

Installer options

The Windows installer is built by CPack using NSIS and supports the default NSIS options for silent installation and install directory.

To silently install to C:\fluent-bit directory here is an example:

<installer exe> /S /D=C:\fluent-bit

The uninstaller also supports a silent uninstall using the same /S flag. This can be used for provisioning with automation like Ansible, Puppet, and so on.

Windows service support

Windows services are equivalent to daemons in Unix (long-running background processes). For v1.5.0 and later, Fluent Bit has native support for Windows services.

For example, you have the following installation layout:

C:\fluent-bit\
├── conf
│   ├── fluent-bit.conf
│   └── parsers.conf
│   └── plugins.conf
└── bin
    ├── fluent-bit.dll
    └── fluent-bit.exe
    └── fluent-bit.pdb

To register Fluent Bit as a Windows service, execute the following command on at a command prompt. A single space is required after binpath=.

sc.exe create fluent-bit binpath= "\fluent-bit\bin\fluent-bit.exe -c \fluent-bit\conf\fluent-bit.conf"

Fluent Bit can be started and managed as a normal Windows service.

sc.exe start fluent-bit
sc.exe query fluent-bit
SERVICE_NAME: fluent-bit
    TYPE               : 10  WIN32_OWN_PROCESS
    STATE              : 4 Running
    ...

To halt the Fluent Bit service, use the stop command.

sc.exe stop fluent-bit

To start Fluent Bit automatically on boot, execute the following:

sc.exe config fluent-bit start= auto

FAQs

Can you manage Fluent Bit service using PowerShell?

Instead of sc.exe, PowerShell can be used to manage Windows services.

Create a Fluent Bit service:

New-Service fluent-bit -BinaryPathName "`"C:\Program Files\fluent-bit\bin\fluent-bit.exe`" -c `"C:\Program Files\fluent-bit\conf\fluent-bit.conf`"" -StartupType Automatic -Description "This service runs Fluent Bit, a log collector that enables real-time processing and delivery of log data to centralized logging systems."

Start the service:

Start-Service fluent-bit

Query the service status:

get-Service fluent-bit | format-list
Name                : fluent-bit
DisplayName         : fluent-bit
Status              : Running
DependentServices   : {}
ServicesDependedOn  : {}
CanPauseAndContinue : False
CanShutdown         : False
CanStop             : True
ServiceType         : Win32OwnProcess

Stop the service:

Stop-Service fluent-bit

Remove the service (requires PowerShell 6.0 or later)

Remove-Service fluent-bit

Compile from source

If you need to create a custom executable, use the following procedure to compile Fluent Bit by yourself.

Preparation

1. Install Microsoft Visual C++ to compile Fluent Bit. You can install the minimum toolkit using the following command:

wget -o vs.exe https://aka.ms/vs/16/release/vs_buildtools.exe
start vs.exe

1. Choose C++ Build Tools and C++ CMake tools for Windows and wait until the process finishes.

2. Install flex and bison. One way to install them on Windows is to use winflexbison.

   Copy

   wget -o winflexbison.zip https://github.com/lexxmark/winflexbison/releases/download/v2.5.22/win_flex_bison-2.5.22.zip
   Expand-Archive winflexbison.zip -Destination C:\WinFlexBison
   cp -Path C:\WinFlexBison\win_bison.exe C:\WinFlexBison\bison.exe
   cp -Path C:\WinFlexBison\win_flex.exe C:\WinFlexBison\flex.exe

3. Add the path C:\WinFlexBison to your systems environment variable Path. .

4. Install OpenSSL binaries, at least the library files and headers.

5. Install to pull the source code from the repository.

Compilation

1. Open the Start menu on Windows and type command Prompt for VS. From the result list, select the one that corresponds to your target system ( x86 or x64).

2. Verify the installed OpenSSL library files match the selected target. You can examine the library files by using the dumpbin command with the /headers option.

3. Clone the source code of Fluent Bit.

4. Compile the source code.

Now you should be able to run Fluent Bit:

fluent-bit.exe -i dummy -o stdout

Packaging

To create a ZIP package, call cpack as follows:

cpack -G ZIP

Fluent Bit v5.0 adds new inputs and processors, expands authentication and TLS options, and standardizes configuration for HTTP-based plugins. It also delivers an important round of performance and scalability work, especially for pipelines that ingest logs, metrics, and traces through HTTP-based protocols. This page gives a quick user-focused overview of the main changes since Fluent Bit v4.2.

For migration-impacting changes, see Upgrade notes.

Performance and scalability

Unified processing and delivery model

Fluent Bit v5.0 continues the move toward a more unified runtime for logs, metrics, and traces. In practice, this means the same core engine improvements benefit more of the pipeline, instead of individual signal paths evolving separately.

For end users, the result is a more consistent behavior across telemetry types and a better base for high-throughput pipelines that mix logs, metrics, and traces in the same deployment.

Refactored HTTP stack

One of the most important v5.0 changes is the refactoring of the HTTP listener stack used by several input plugins. Fluent Bit now uses a shared HTTP server implementation across the major HTTP-based receivers instead of maintaining separate code paths.

This work improves:

• concurrency through shared listener worker support

• consistency of request handling across HTTP-based inputs

• buffer enforcement and connection handling

• maintainability, which reduces drift between plugin implementations

The biggest user-facing beneficiaries are:

• HTTP input

• Splunk input

• Elasticsearch input

• OpenTelemetry input

If you run large HTTP or OTLP ingestion workloads, v5.0 is not only a feature release. It is also a meaningful runtime improvement.

Configuration and operations

Shared HTTP listener settings

HTTP-based inputs now use a shared listener configuration model. The preferred setting names are:

• http_server.http2

• http_server.buffer_chunk_size

• http_server.buffer_max_size

• http_server.max_connections

• http_server.workers

• http_server.ingress_queue_event_limit

• http_server.ingress_queue_byte_limit

Legacy aliases such as http2, buffer_chunk_size, and buffer_max_size still work, but new configurations should use the http_server.* names.

Affected plugin families include:

• HTTP input

• Splunk input

• Elasticsearch input

• OpenTelemetry input

Mutual TLS for inputs

Input plugins that support TLS can now require client certificate verification with tls.verify_client_cert. This makes it easier to run mutual TLS (mTLS) directly on Fluent Bit listeners.

See TLS.

JSON health endpoint in API v2

The built-in HTTP server exposes /api/v2/health, which returns health status as JSON and uses the HTTP status code to indicate healthy (200) or unhealthy (500) state.

See Monitoring.

Inputs

New fluentbit_logs input

The Fluent Bit logs input routes Fluent Bit internal logs back into the pipeline as structured records. This lets you forward agent diagnostics to any supported destination.

HTTP input remote address capture

The HTTP input adds:

• add_remote_addr

• remote_addr_key

These settings let you attach the client address from X-Forwarded-For to each ingested record.

OAuth 2.0 bearer token validation on HTTP-based inputs

HTTP-based receivers can validate incoming bearer tokens with:

• oauth2.validate

• oauth2.issuer

• oauth2.jwks_url

• oauth2.allowed_audience

• oauth2.allowed_clients

• oauth2.jwks_refresh_interval

This is available on the relevant input plugins, including HTTP and OpenTelemetry.

OpenTelemetry input improvements

The OpenTelemetry input in v5.0 expands user-visible behavior with:

• shared HTTP listener worker support

• OAuth 2.0 bearer token validation

• stable JSON metrics ingestion over OTLP/HTTP

• improved JSON trace validation and error reporting

Kubernetes events state database controls

The Kubernetes events input documents additional SQLite controls:

• db.journal_mode

• db.locking

These settings help tune event cursor persistence and database access behavior.

Processors

New cumulative-to-delta processor

The cumulative to delta processor converts cumulative monotonic metrics to delta values, which is useful when scraping Prometheus-style metrics but exporting to backends that expect deltas.

New topological data analysis processor

The topological data analysis processor adds a metrics processor for topology-based analysis workflows.

Sampling processor updates

The sampling processor adds legacy_reconcile for tail sampling, which helps compare the optimized reconciler with the previous behavior when validating upgrades.

Outputs

HTTP output OAuth 2.0 client credentials

The HTTP output now supports built-in OAuth 2.0 client credentials with:

• basic

• post

• private_key_jwt

You can configure token acquisition directly in Fluent Bit with the oauth2.* settings.

More compression options for cloud outputs

Several outputs gained additional compression support in the v4.2 to v5.0 range:

• Amazon Kinesis Data Streams: gzip, zstd, snappy

• Amazon Kinesis Data Firehose: snappy added alongside existing codecs

• : snappy added alongside existing codecs

• : zstd support for transfer compression

Monitoring changes

fluentbit_hot_reloaded_times is now a counter

The fluentbit_hot_reloaded_times metric changed from a gauge to a counter, which makes it safe to use with PromQL functions such as rate() and increase().

New output backpressure visibility

v5.0 adds output backpressure duration metrics so you can observe time spent waiting because of downstream pressure.

See Monitoring.

The following article covers the relevant compatibility changes for users upgrading from previous Fluent Bit versions.

For more details about changes on each release, refer to the Official Release Notes.

Release notes will be prepared in advance of a Git tag for a release. An official release should provide both a tag and a release note together to allow users to verify and understand the release contents.

The tag drives the binary release process. Release binaries (containers and packages) will appear after a tag and its associated release note. This lets users to expect the new release binary to appear and allow/deny/update it as appropriate in their infrastructure.

Fluent Bit v5.0

hot_reloaded_times metric type change

The internal metric fluentbit_hot_reloaded_times has changed from a gauge to a counter. The previous gauge registration caused incorrect results when using PromQL functions like rate() and increase(), which expect counters.

If you have Prometheus dashboards or alerting rules that reference fluentbit_hot_reloaded_times, update them to use counter-appropriate PromQL functions (for example, rate() or increase() instead of gauge-specific functions like delta()).

Shared HTTP listener settings for HTTP-based inputs

The HTTP-based input plugins now use a shared HTTP listener configuration model. In Fluent Bit v5.0, the canonical setting names are:

• http_server.http2

• http_server.buffer_chunk_size

• http_server.buffer_max_size

• http_server.max_connections

• http_server.workers

• http_server.ingress_queue_event_limit

• http_server.ingress_queue_byte_limit

Legacy per-plugin names such as http2, buffer_chunk_size, and buffer_max_size are still accepted as compatibility aliases, but new configurations should use the http_server.* names.

If you tune http, splunk, elasticsearch, opentelemetry, or prometheus_remote_write inputs, review those sections and migrate to the shared naming so future upgrades are clearer.

Mutual TLS for input plugins

Input plugins that support TLS now also support tls.verify_client_cert. Enable this option to require and validate the client certificate presented by the sender.

If you terminate TLS directly in Fluent Bit and need mutual TLS (mTLS), add tls.verify_client_cert on together with the usual tls.crt_file and tls.key_file settings.

New internal logs input

Fluent Bit v5.0 adds the fluentbit_logs input plugin, which mirrors the Fluent Bit internal log stream back into the data pipeline as structured log records.

Use this input if you want to forward Fluent Bit diagnostics to another destination, filter them, or store them alongside the rest of your telemetry.

Emitter backpressure with filesystem storage

The internal emitter plugin, used by filters such as rewrite_tag, now automatically enables storage.pause_on_chunks_overlimit when filesystem storage is in use and that option hasn't been explicitly configured.

Previously, the emitter could accumulate chunks beyond the storage.max_chunks_up limit. Pipelines that use rewrite_tag or other emitter-backed filters with filesystem storage will now pause when the configured storage limit is reached.

If you rely on the previous unlimited accumulation behavior, explicitly set storage.pause_on_chunks_overlimit off on the relevant input. Otherwise, review your storage.max_chunks_up value to ensure it's tuned for your expected throughput.

More OAuth 2.0 coverage

Fluent Bit v5.0 expands OAuth 2.0 support in both directions:

• HTTP-based inputs can validate incoming bearer tokens using oauth2.validate, oauth2.issuer, and oauth2.jwks_url.

• The HTTP output can acquire access tokens with oauth2.enable and supports basic, post, and private_key_jwt client authentication.

If you previously handled authentication outside Fluent Bit for these cases, review the plugin pages for the new built-in options.

Forward output: retain_metadata_in_forward_mode default changed to true

Starting in Fluent Bit v5.0.4, the forward output plugin's retain_metadata_in_forward_mode option defaults to true (previously false). When true, Fluent Bit embeds event metadata into the Forward protocol payload using the extended MessagePack format.

Fluentd receivers don't understand this format and will reject events with a warning similar to:

[input1] skip invalid event: host="..." tag="..." time=[..., {}] record={...}

If you send data from Fluent Bit to a Fluentd aggregator using the forward output, explicitly set retain_metadata_in_forward_mode false to restore the previous behavior:

pipeline:
  outputs:
    - name: forward
      match: "*"
      host: fluentd-host
      port: 24224
      retain_metadata_in_forward_mode: false

[OUTPUT]
    Name    forward
    Match   *
    Host    fluentd-host
    Port    24224
    Retain_Metadata_In_Forward_Mode false

For more details, see GitHub issue #11877.

For a broader overview of user-visible additions in this release, see What's new in Fluent Bit v5.0.

Fluent Bit v4.2

Vivo exporter output plugin

The HTTP endpoint paths exposed by the Vivo exporter output plugin have changed. All endpoints now follow an /api/v1/ prefix:

If you have tooling or dashboards that query the Vivo exporter HTTP endpoints directly, update the endpoint paths accordingly.

Fluent Bit v4.0

Package support for older Linux distributions

Official binary packages are no longer produced for the following Linux distributions:

• Ubuntu 16.04 (Xenial)

• Ubuntu 18.04 (Bionic)

• Ubuntu 20.04 (Focal)

Users on these platforms should upgrade their OS or build Fluent Bit from source.

Kafka plugin support on older platforms

The Kafka input and output plugins are disabled in official packages for CentOS 7 and Amazon Linux 2 (ARM64). This is due to a Kafka library (librdkafka) update that requires a newer glibc version than these platforms provide.

Users who need Kafka support on these platforms must build Fluent Bit from source against a compatible version of librdkafka.

Fluent Bit v3.0

HTTP/2 enabled by default for HTTP-based input plugins

The following input plugins now have HTTP/2 support enabled by default (http2 true):

• opentelemetry

• splunk

• elasticsearch

• http

These plugins transparently support both HTTP/1.1 and HTTP/2 connections. If your clients don't support HTTP/2, or if you have a reverse proxy or load balancer that doesn't handle HTTP/2 correctly, add http2 off to the affected input plugin configuration section.

Fluent Bit v2.0

TLS library

mbedTLS is no longer supported as a TLS backend. All TLS connections now use OpenSSL. If you compile Fluent Bit from source and previously linked against mbedTLS, you must now link against OpenSSL. Official binary packages already use OpenSSL.

Fluent Bit v1.9.9

The td-agent-bit package is no longer provided after this release. Users should switch to the fluent-bit package.

Fluent Bit v1.6

If you are migrating from previous version of Fluent Bit, review the following important changes:

Tail input plugin

By default, the tail input plugin follows a file from the end after the service starts, instead of reading it from the beginning. Every file found when the plugin starts is followed from it last position. New files discovered at runtime or when files rotate are read from the beginning.

To keep the old behavior, set the option read_from_head to true.

Stackdriver output plugin

The project_id of resource in LogEntry sent to Google Cloud Logging would be set to the project_id rather than the project number. To learn the difference between Project ID and project number, see Creating and managing projects.

If you have existing queries based on the resource's project_id, update your query accordingly.

Fluent Bit v1.5

The migration from v1.4 to v1.5 is pretty straightforward.

• The keepalive configuration mode has been renamed to net.keepalive. Now, all Network I/O keepalive is enabled by default. To learn more about this and other associated configuration properties read the Networking Administration section. - If you use the Elasticsearch output plugin, the default value of type changed from flb_type to _doc. Many versions of Elasticsearch tolerate this, but Elasticsearch v5.6 through v6.1 require a type without a leading underscore. See the Elasticsearch output plugin documentation FAQ entry for more.

Fluent Bit v1.4

If you are migrating from Fluent Bit v1.3, there are no breaking changes.

Fluent Bit v1.3

If you are migrating from Fluent Bit v1.2 to v1.3, there are no breaking changes. If you are upgrading from an older version, review the following incremental changes:

Fluent Bit v1.2

Docker, JSON, parsers, and decoders

Fluent Bit v1.2 fixed many issues associated with JSON encoding and decoding.

For example, when parsing Docker logs, it's no longer necessary to use decoders. The new Docker parser looks like this:

[PARSER]
  Name        docker
  Format      json
  Time_Key    time
  Time_Format %Y-%m-%dT%H:%M:%S.%L
  Time_Keep   On

Kubernetes filter

Fluent Bit made improvements to Kubernetes Filter handling of string-encoded log messages. If the Merge_Log option is enabled, it will try to handle the log content as a JSON map, if so, it will add the keys to the root map.

In addition, fixes and improvements were made to the Merge_Log_Key option. If a merge log succeed, all new keys will be packaged under the key specified by this option. A suggested configuration is as follows:

[FILTER]
  Name Kubernetes
  Match kube.*
  Kube_Tag_Prefix kube.var.log.containers.
  Merge_Log On
  Merge_Log_Key log_processed

As an example, if the original log content is the following map:

{"key1": "val1", "key2": "val2"}

the final record will be composed as follows:

{"log": "{\"key1\": \"val1\", \"key2\": \"val2\"}", "log_processed": { "key1": "val1", "key2": "val2" } }

Fluent Bit v1.1

If you are upgrading from Fluent Bit 1.0.x or earlier, review the following relevant changes when switching to Fluent Bit v1.1 or later series:

Kubernetes filter

Fluent Bit introduced a new configuration property called Kube_Tag_Prefix to help Tag prefix resolution and address an unexpected behavior in previous versions.

During the 1.0.x release cycle, a commit in the Tail input plugin changed the default behavior on how the Tag was composed when using the wildcard for expansion generating breaking compatibility with other services. Consider the following configuration example:

[INPUT]
  Name tail
  Path /var/log/containers/*.log
  Tag kube.*

The expected behavior is that Tag will be expanded to:

text kube.var.log.containers.apache.log

The change introduced in the 1.0 series switched from absolute path to the base filename only:

text kube.apache.log

THe Fluent Bit v1.1 release restored the default behavior and now the Tag is composed using the absolute path of the monitored file.

Having absolute path in the Tag is relevant for routing and flexible configuration where it also helps to keep compatibility with Fluentd behavior.

This behavior switch in Tail input plugin affects how Filter Kubernetes operates. When the filter is used it needs to perform local metadata lookup that comes from the file names when using Tail as a source. With the new Kube_Tag_Prefix option you can specify the prefix used in the Tail input plugin. For the previous configuration example the new configuration will look like:

[INPUT]
  Name tail
  Path /var/log/containers/*.log
  Tag kube.*

[FILTER]
  Name kubernetes
  Match *
  Kube_Tag_Prefix kube.var.log.containers.

The proper value for Kube_Tag_Prefix must be composed by Tag prefix set in Tail input plugin plus the converted monitored directory replacing slashes with dots.

In Fluent Bit v3.2 and later, YAML configuration files support all of the settings and features that classic configuration files support, plus additional features that classic configuration files don't support, like processors.

YAML configuration files support the following top-level sections:

• env: Configures environment variables.

• includes: Specifies additional YAML configuration files to include as part of a parent file.

• service: Configures global properties of the Fluent Bit service.

• pipeline: Configures active inputs, filters, and outputs.

• parsers: Defines custom parsers.

• multiline_parsers: Defines custom multiline parsers.

• plugins: Defines paths for custom plugins.

• upstream_servers: Defines nodes for output plugins.

The pipeline section of YAML configuration files defines the flow of how data is collected, processed, and sent to its final destination. This section contains the following subsections:

• inputs: Configures input plugins.

• filters: Configures filters.

• outputs: Configures output plugins.

Syntax

The pipeline section of a YAML configuration file uses the following syntax:

pipeline:
    inputs:
        ...
    filters:
        ...
    outputs:
        ...

Each of the subsections for inputs, filters, and outputs constitutes an array of maps that has the parameters for each. Most properties are either strings or numbers and can be defined directly.

For example:

pipeline:
    inputs:
        - name

Inputs

The inputs section defines one or more input plugins. In addition to the settings unique to each plugin, all input plugins support the following configuration parameters:

The name parameter is required and defines for Fluent Bit which input plugin should be loaded. The tag parameter is required for all plugins except for the forward plugin, which provides dynamic tags.

There is no hard-coded limit on the number of input plugins. The practical maximum depends on available system resources such as memory and file descriptors.

Shared HTTP listener settings for inputs

Some HTTP-based input plugins share the same listener implementation and support the following common settings in addition to their plugin-specific parameters:

These settings are shared by HTTP-based inputs such as http, splunk, elasticsearch, opentelemetry, and prometheus_remote_write. Use these keys the same way across those plugins.

If a plugin page shows one of the http_server.* keys in its configuration table, it is documenting one of these shared listener settings, not a plugin-specific behavior.

When http_server.workers is 1, Fluent Bit does not use the deferred ingress queue, so the two http_server.ingress_queue_* settings have no effect.

For backward compatibility, some plugins also accept the legacy aliases http2, buffer_max_size, buffer_chunk_size, max_connections, and workers.

Incoming OAuth 2.0 JWT validation settings

The HTTP-based input plugins that support bearer token validation share the following oauth2.* settings:

When validation is enabled, requests without a valid Authorization: Bearer <token> header are rejected.

Example input configuration

The following is an example of an inputs section that contains a cpu plugin.

pipeline:
    inputs:
        - name: cpu
          tag: my_cpu

Filters

The filters section defines one or more filters. In addition to the settings unique to each filter, all filters support the following configuration parameters:

The name parameter is required and lets Fluent Bit know which filter should be loaded. One of either the match or match_regex parameters is required. If both are specified, match_regex takes precedence.

There is no hard-coded limit on the number of filter plugins. The practical maximum depends on available system resources such as memory.

Example filter configuration

The following is an example of a filters section that contains a grep plugin:

pipeline:
    filters:
        - name: grep
          match: '*'
          regex: log aa

Outputs

The outputs section defines one or more output plugins. In addition to the settings unique to each plugin, all output plugins support the following configuration parameters:

Fluent Bit has no hard-coded limit on the number of output plugins. The routing bitmask is dynamically sized at startup based on the number of configured output plugins. The practical maximum depends on available system resources such as memory and file descriptors.

Outgoing OAuth 2.0 client credentials settings

Output plugins that support outgoing OAuth 2.0 authentication can expose the following shared oauth2.* settings:

Example output configuration

The following is an example of an outputs section that contains a stdout plugin:

pipeline:
    outputs:
        - name: stdout
          match: 'my*cpu'

One of the ways to configure Fluent Bit is using a main configuration file. Fluent Bit allows the use one configuration file that works at a global scope and uses the defined Format and Schema.

The main configuration file supports four sections:

• Service

• Input

• Filter

• Output

It's also possible to split the main configuration file into multiple files using the Include File feature to include external files.

Service

The Service section defines global properties of the service. The following keys are:

The following is an example of a SERVICE section:

[SERVICE]
  Flush           5
  Daemon          off
  Log_Level       debug

For scheduler and retry details, see scheduling and retries.

Config input

The INPUT section defines a source (related to an input plugin). Each input plugin can add its own configuration keys:

Name is mandatory and tells Fluent Bit which input plugin to load. Tag is mandatory for all plugins except for the input forward plugin, which provides dynamic tags.

There is no hard-coded limit on the number of INPUT sections. The practical maximum depends on available system resources such as memory and file descriptors.

Example

The following is an example of an INPUT section:

[INPUT]
  Name cpu
  Tag  my_cpu

Config filter

The FILTER section defines a filter (related to an filter plugin). Each filter plugin can add it own configuration keys. The base configuration for each FILTER section contains: