In 2014, the Fluentd team at Treasure Data was forecasting the need for a lightweight log processor for constraint environments like embedded Linux and gateways. The project aimed to be part of the Fluentd ecosystem. At that moment, Eduardo Silva created Fluent Bit, a new open source solution, written from scratch and available under the terms of the Apache License v2.0.

After the project matured, it gained traction for normal Linux systems. With the new containerized world, the Cloud Native community asked to extend the project scope to support more sources, filters, and destinations. Not long after, Fluent Bit became one of the preferred solutions to solve the logging challenges in Cloud environments.

Fluent Bit is an open source telemetry agent specifically designed to efficiently handle the challenges of collecting and processing telemetry data across a wide range of environments, from constrained systems to complex cloud infrastructures. Managing telemetry data from various sources and formats can be a constant challenge, particularly when performance is a critical factor.

Rather than serving as a drop-in replacement, Fluent Bit enhances the observability strategy for your infrastructure by adapting and optimizing your existing logging layer, and adding metrics and traces processing. Fluent Bit supports a vendor-neutral approach, seamlessly integrating with other ecosystems such as Prometheus and OpenTelemetry. Trusted by major cloud providers, banks, and companies in need of a ready-to-use telemetry agent solution, Fluent Bit effectively manages diverse data sources and formats while maintaining optimal performance and keeping resource consumption low.

Fluent Bit can be deployed as an edge agent for localized telemetry data handling or utilized as a central aggregator/collector for managing telemetry data across multiple sources and environments.

When Fluent Bit processes data, it uses the system memory (heap) as a primary and temporary place to store the record logs before they get delivered. The records are processed in this private memory area.

Buffering is the ability to store the records, and continue storing incoming data while previous data is processed and delivered. Buffering in memory is the fastest mechanism, but there are scenarios requiring special strategies to deal with backpressure, data safety, or to reduce memory consumption by the service in constrained environments.

Network failures or latency in third party service is common. When data can't be delivered fast enough and new data to process arrives, the system can face backpressure.

Fluent Bit buffering strategies are designed to solve problems associated with backpressure and general delivery failures. Fluent Bit offers a primary buffering mechanism in memory and an optional secondary one using the file system. With this hybrid solution you can accommodate any use case safely and keep a high performance while processing your data.

These mechanisms aren't mutually exclusive. When data is ready to be processed or delivered it's always be in memory, while other data in the queue might be in the file system until is ready to be processed and moved up to memory.

To learn more about the buffering configuration in Fluent Bit, see Buffering & Storage.

Before diving into Fluent Bit you might want to get acquainted with some of the key concepts of the service. This document provides an introduction to those concepts and common Fluent Bit terminology. Reading this document will help you gain a more general understanding of the following topics:

• Event or Record

• Filtering

• Tag

• Timestamp

• Match

• Structured Message

Event or Record

Every incoming piece of data that belongs to a log or a metric that's retrieved by Fluent Bit is considered an Event or a Record.

As an example, consider the following content of a Syslog file:

Jan 18 12:52:16 flb systemd[2222]: Starting GNOME Terminal Server
Jan 18 12:52:16 flb dbus-daemon[2243]: [session uid=1000 pid=2243] Successfully activated service 'org.gnome.Terminal'
Jan 18 12:52:16 flb systemd[2222]: Started GNOME Terminal Server.
Jan 18 12:52:16 flb gsd-media-keys[2640]: # watch_fast: "/org/gnome/terminal/legacy/" (establishing: 0, active: 0)

It contains four lines that represent four independent Events.

An Event is comprised of:

• timestamp

• key/value metadata (v2.1.0 and greater)

• payload

Event format

The Fluent Bit wire protocol represents an Event as a two-element array with a nested array as the first element:

[[TIMESTAMP, METADATA], MESSAGE]

where

• TIMESTAMP is a timestamp in seconds as an integer or floating point value (not a string).

• METADATA is an object containing event metadata, and might be empty.

• MESSAGE is an object containing the event body.

Fluent Bit versions prior to v2.1.0 used:

[TIMESTAMP, MESSAGE]

to represent events. This format is still supported for reading input event streams.

Filtering

You might need to perform modifications on an Event's content. The process to alter, append to, or drop Events is called filtering.

Use filtering to:

• Append specific information to the Event like an IP address or metadata.

• Select a specific piece of the Event content.

• Drop Events that match a certain pattern.

Tag

Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string used in a later stage by the Router to decide which Filter or Output phase it must go through.

Most tags are assigned manually in the configuration. If a tag isn't specified, Fluent Bit assigns the name of the Input plugin instance where that Event was generated from.

A tagged record must always have a Matching rule. To learn more about Tags and Matches, see Routing.

Timestamp

The timestamp represents the time an Event was created. Every Event contains an associated timestamps. All events have timestamps, and they're set by the input plugin or discovered through a data parsing process.

The timestamp is a numeric fractional integer in the format:

SECONDS.NANOSECONDS

where:

• _SECONDS_ is the number of seconds that have elapsed since the Unix epoch.

• _NANOSECONDS_ is a fractional second or one thousand-millionth of a second.

Match

Fluent Bit lets you route your collected and processed Events to one or multiple destinations. A Match represents a rule to select Events where a Tag matches a defined rule.

To learn more about Tags and Matches, see Routing.

Structured messages

Source events can have a structure. A structure defines a set of keys and values inside the Event message to implement faster operations on data modifications. Fluent Bit treats every Event message as a structured message.

Consider the following two messages:

• No structured message

  Copy

  "Project Fluent Bit created on 1398289291"

• With a structured message

  Copy

  {"project": "Fluent Bit", "created": 1398289291}

For performance reasons, Fluent Bit uses a binary serialization data format called MessagePack.

Fluent Bit has very low CPU and memory consumption. It's compatible with most x86-, x86_64-, arm32v7-, and arm64v8-based platforms.

The build process requires the following components:

• Compiler: GCC or clang

• CMake

• Flex and Bison: Required for Stream Processor or Record Accessor

• Libyaml development headers and libraries

Core has no other dependencies. Some features depend on third-party components. For example, output plugins with special backend libraries like Kafka include those libraries in the main source code repository.

Fluent Bit is supported on Linux on IBM Z(s390x), but the WASM and LUA filter plugins aren't.

Telemetry data processing can be complex, especially at scale. That's why Fluentd was created. Fluentd is more than a simple tool, it's grown into a fullscale ecosystem that contains SDKs for different languages and subprojects like Fluent Bit.

Here, we describe the relationship between the Fluentd and Fluent Bit open source projects.

Both projects are:

• Licensed under the terms of Apache License v2.0.

• Graduated hosted projects by the Cloud Native Computing Foundation (CNCF).

• Production grade solutions: Deployed millions of times every single day.

• Vendor neutral and community driven.

• Widely adopted by the industry: Trusted by major companies like AWS, Microsoft, Google Cloud, and hundreds of others.

The projects have many similarities: Fluent Bit is designed and built on top of the best ideas of Fluentd architecture and general design. Which one you choose depends on your end-users' needs.

The following table describes a comparison of different areas of the projects:

Both Fluentd and Fluent Bit can work as Aggregators or Forwarders, and can complement each other or be used as standalone solutions.

In the recent years, cloud providers have switched from Fluentd to Fluent Bit for performance and compatibility. Fluent Bit is now considered the next-generation solution.

Fluent Bit supports the following operating systems and architectures:

From an architecture support perspective, Fluent Bit is fully functional on x86_64, Arm64v8, and Arm32v7 based processors.

Fluent Bit can work also on macOS and Berkeley Software Distribution (BSD) systems, but not all plugins will be available on all platforms.

Official support is based on community demand. Fluent Bit might run on older operating systems, but must be built from source, or using custom packages from enterprise providers.

Fluent Bit is supported for Linux on IBM Z (s390x) environments with some restrictions, but only container images are provided for these targets officially.

Fluent Bit is a fast and lightweight telemetry agent for logs, metrics, and traces for Linux, macOS, Windows, and BSD family operating systems. Fluent Bit has been made with a strong focus on performance to allow the collection and processing of telemetry data from different sources without complexity.

Features

• High performance: High throughput with low resources consumption

• Data parsing

  • Convert your unstructured messages using our parsers: JSON, Regex, LTSV and Logfmt

• Metrics support: Prometheus and OpenTelemetry compatible

• Reliability and data integrity

  • Backpressure handling

  • Data buffering in memory and file system

• Networking

  • Security: Built-in TLS/SSL support

  • Asynchronous I/O

• Pluggable architecture and extensibility: Inputs, Filters and Outputs:

  • Connect nearly any source to nearly any destination using preexisting plugins

  • Extensibility:

    • Write input, filter, or output plugins in the C language

    • WASM: WASM Filter Plugins or WASM Input Plugins

    • Write Filters in Lua or Output plugins in Golang

• Monitoring: Expose internal metrics over HTTP in JSON and Prometheus format

• Stream Processing: Perform data selection and transformation using simple SQL queries

  • Create new streams of data using query results

  • Aggregation windows

  • Data analysis and prediction: Timeseries forecasting

• Portable: Runs on Linux, macOS, Windows and BSD systems

Fluent Bit, Fluentd and CNCF

Fluent Bit is a CNCF graduated sub-project under the umbrella of Fluentd. Fluent Bit is licensed under the terms of the Apache License v2.0.

Fluent Bit was originally created by Eduardo Silva and is now sponsored by Chronosphere. As a CNCF-hosted project, it is a fully vendor-neutral and community-driven project.

In production environments you need full control of the data you're collecting. Filtering lets you alter the collected data before delivering it to a destination.

Filtering is implemented through plugins. Each available filter can be used to match, exclude, or enrich your logs with specific metadata.

Fluent Bit support many filters. A common use case for filtering is Kubernetes deployments. Every pod log needs the proper metadata associated with it.

Like input plugins, filters run in an instance context, which has its own independent configuration. Configuration keys are often called properties.

For more details about the Filters available and their usage, see Filters.

Fluent Bit provides input plugins to gather information from different sources. Some plugins collect data from log files, while others can gather metrics information from the operating system. There are many plugins to suit different needs.

When an input plugin loads, an internal instance is created. Each instance has its own independent configuration. Configuration keys are often called properties.

Every input plugin has its own documentation section that specifies how to use it and what properties are available.

For more details, see Input Plugins.

The buffer phase in the pipeline aims to provide a unified and persistent mechanism to store your data, using the primary in-memory model or the file system-based mode.

The buffer phase contains the data in an immutable state, meaning that no other filter can be applied.

Buffered data uses the Fluent Bit internal binary representation, which isn't raw text.

Fluent Bit offers a buffering mechanism in the file system that acts as a backup system to avoid data loss in case of system failures.

Dealing with raw strings or unstructured messages is difficult. Having a structure makes data more usable. Set a structure to the incoming data by using input plugins as data is collected:

The parser converts unstructured data to structured data. As an example, consider the following Apache (HTTP Server) log entry:

192.168.2.20 - - [28/Jul/2006:10:27:10 -0300] "GET /cgi-bin/try/ HTTP/1.0" 200 3395

This log line is a raw string without format. Structuring the log makes it easier to process the data later. If the regular expression parser is used, the log entry could be converted to:

{
  "host":    "192.168.2.20",
  "user":    "-",
  "method":  "GET",
  "path":    "/cgi-bin/try/",
  "code":    "200",
  "size":    "3395",
  "referer": "",
  "agent":   ""
 }

Parsers are fully configurable and are independently and optionally handled by each input plugin. For more details, see Parsers.

Routing is a core feature that lets you route your data through filters and then to one or multiple destinations. The router relies on the concept of Tags and Matching rules.

There are two important concepts in Routing:

• Tag

• Match

When data is generated by an input plugin, it comes with a Tag. A Tag is a human-readable indicator that helps to identify the data source. Tags are usually configured manually.

To define where to route data, specify a Match rule in the output configuration.

Consider the following configuration example that delivers CPU metrics to an Elasticsearch database and Memory (mem) metrics to the standard output interface:

[INPUT]
    Name cpu
    Tag  my_cpu

[INPUT]
    Name mem
    Tag  my_mem

[OUTPUT]
    Name   es
    Match  my_cpu

[OUTPUT]
    Name   stdout
    Match  my_mem

Routing reads the Input Tag and the Output Match rules. If data has a Tag that doesn't match at routing time, the data is deleted.

Routing with Wildcard

Routing is flexible enough to support wildcards in the Match pattern. The following example defines a common destination for both sources of data:

[INPUT]
    Name cpu
    Tag  my_cpu

[INPUT]
    Name mem
    Tag  my_mem

[OUTPUT]
    Name   stdout
    Match  my_*

The match rule is set to my_*, which matches any Tag starting with my_*.

Routing with Regex

Routing also provides support for regular expressions with the Match_Regex pattern, allowing for more complex and precise matching criteria. The following example demonstrates how to route data from sources based on a regular expression:

[INPUT]
    Name temperature_sensor
    Tag  temp_sensor_A

[INPUT]
    Name humidity_sensor
    Tag  humid_sensor_B

[OUTPUT]
    Name         stdout
    Match_regex  .*_sensor_[AB]

In this configuration, the Match_regex rule is set to .*_sensor_[AB]. This regular expression matches any Tag that ends with _sensor_A or _sensor_B, regardless of what precedes it. This approach provides a more flexible and powerful way to handle different source tags with a single routing rule.

The output interface lets you define destinations for your data. Common destinations are remote services, local file systems, or other standard interfaces. Outputs are implemented as plugins.

When an output plugin is loaded, an internal instance is created. Every instance has its own independent configuration. Configuration keys are often called properties.

Every output plugin has its own documentation section specifying how it can be used and what properties are available.

For more details, see .

uses as its build system.

Requirements

• CMake 3.12 or greater. You might need to use cmake3 instead of cmake.

• Flex

• Bison 3 or greater

• YAML headers

• OpenSSL headers

Prepare environment

If you already know how CMake works, you can skip this section and review the available .

The following steps explain how to build and install the project with the default options.

1. Change to the build/ directory inside the Fluent Bit sources:

2. Let configure the project specifying where the root path is located:

   This command displays a series of results similar to:

3. Start the compilation process using the make command:

   This command displays results similar to:

4. To continue installing the binary on the system, use make install:

   If the command indicates insufficient permissions, prefix the command with sudo.

Build options

Fluent Bit provides configurable options to CMake that can be enabled or disabled.

General options

Development options

Optimization options

Input plugins

Input plugins gather information from a specific source type like network interfaces, some built-in metrics, or through a specific input device. The following input plugins are available:

Filter plugins

Filter plugins let you modify, enrich or drop records. The following table describes the filters available on this version:

Output plugins

Output plugins let you flush the information to some external interface, service, or terminal. The following table describes the output plugins available:

Processor plugins

Processor plugins handle the events within the processor pipelines to allow modifying, enriching, or dropping events.

The following table describes the processors available:

Install on Amazon Linux

Fluent Bit is distributed as fluent-bit package and is available for the latest Amazon Linux 2 and Amazon Linux 2023. The following architectures are supported

• x86_64

• aarch64 / arm64v8

Single line install

A simple installation script is provided to be used for most Linux targets. This will always install the most recent version released.

This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to follow the instructions below.

Amazon Linux 2022

Amazon Linux 2022 was previously supported but is removed since it became GA Amazon Linux 2023

Configure Yum

We provide fluent-bit through a Yum repository. In order to add the repository reference to your system, please add a new file called fluent-bit.repo in /etc/yum.repos.d/ with the following content:

Amazon Linux 2

Amazon Linux 2023

Note: we encourage you always enable the gpgcheck for security reasons. All our packages are signed.

Updated key from March 2022

The GPG Key fingerprint of the new key is:

The GPG Key fingerprint of the old key is:

Install

Once your repository is configured, run the following command to install it:

Now the following step is to instruct systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/messages file.

The most secure option is to create the repositories according to the instructions for your specific OS.

A simple installation script is provided to be used for most Linux targets. This will by default install the most recent version released.

This is purely a convenience helper and should always be validated prior to use.

GPG key updates

From the 1.9.0 and 1.8.15 releases please note that the GPG key has been updated at so ensure this new one is added.

The GPG Key fingerprint of the new key is:

The previous key is still available at and may be required to install previous versions.

The GPG Key fingerprint of the old key is:

Refer to the to see which platforms are supported in each release.

Migration to Fluent Bit

From version 1.9, td-agent-bit is a deprecated package and is removed after 1.9.9. The correct package name to use now is fluent-bit.

, including its core, plugins, and tools are distributed under the terms of the :

Container deployment

Install on Linux (packages)

Install on Windows (packages)

Install on macOS (packages)

Compile from Source (Linux, Windows, FreeBSD, macOS)

Sandbox Environment

If you are interested in learning about Fluent Bit you can try out the sandbox environment:

Enterprise Packages

The following article cover the relevant notes for users upgrading from previous Fluent Bit versions. We aim to cover compatibility changes that you must be aware of.

For more details about changes on each release please refer to the .

Note: release notes will be prepared in advance of a Git tag for a release so an official release should provide both a tag and a release note together to allow users to verify and understand the release contents.

The tag drives the overall binary release process so release binaries (containers/packages) will appear after a tag and its associated release note. This allows users to expect the new release binary to appear and allow/deny/update it as appropriate in their infrastructure.

Fluent Bit v1.9.9

The td-agent-bit package is no longer provided after this release. Users should switch to the fluent-bit package.

Fluent Bit v1.6

If you are migrating from previous version of Fluent Bit please review the following important changes:

Tail Input Plugin

Now by default the plugin follows a file from the end once the service starts (old behavior was always read from the beginning). For every file found at start, its followed from it last position, for new files discovered at runtime or rotated, they are read from the beginning.

If you desire to keep the old behavior you can set the option read_from_head to true.

Stackdriver Output Plugin

The project_id of in sent to Google Cloud Logging would be set to the project ID rather than the project number. To learn the difference between Project ID and project number, see for more details.

If you have any existing queries based on the resource's project_id, please update your query accordingly.

Fluent Bit v1.5

The migration from v1.4 to v1.5 is pretty straightforward.

Fluent Bit v1.4

If you are migrating from Fluent Bit v1.3, there are no breaking changes. Just new exciting features to enjoy :)

Fluent Bit v1.3

If you are migrating from Fluent Bit v1.2 to v1.3, there are no breaking changes. If you are upgrading from an older version please review the incremental changes below.

Fluent Bit v1.2

Docker, JSON, Parsers and Decoders

On Fluent Bit v1.2 we have fixed many issues associated with JSON encoding and decoding, for hence when parsing Docker logs is no longer necessary to use decoders. The new Docker parser looks like this:

Note: again, do not use decoders.

Kubernetes Filter

We have done improvements also on how Kubernetes Filter handle the stringified log message. If the option Merge_Log is enabled, it will try to handle the log content as a JSON map, if so, it will add the keys to the root map.

In addition, we have fixed and improved the option called Merge_Log_Key. If a merge log succeed, all new keys will be packaged under the key specified by this option, a suggested configuration is as follows:

As an example, if the original log content is the following map:

the final record will be composed as follows:

Fluent Bit v1.1

If you are upgrading from Fluent Bit <= 1.0.x you should take in consideration the following relevant changes when switching to Fluent Bit v1.1 series:

Kubernetes Filter

We introduced a new configuration property called Kube_Tag_Prefix to help Tag prefix resolution and address an unexpected behavior that landed in previous versions.

During 1.0.x release cycle, a commit in Tail input plugin changed the default behavior on how the Tag was composed when using the wildcard for expansion generating breaking compatibility with other services. Consider the following configuration example:

The expected behavior is that Tag will be expanded to:

but the change introduced in 1.0 series switched from absolute path to the base file name only:

On Fluent Bit v1.1 release we restored to our default behavior and now the Tag is composed using the absolute path of the monitored file.

Having absolute path in the Tag is relevant for routing and flexible configuration where it also helps to keep compatibility with Fluentd behavior.

This behavior switch in Tail input plugin affects how Filter Kubernetes operates. As you know when the filter is used it needs to perform local metadata lookup that comes from the file names when using Tail as a source. Now with the new Kube_Tag_Prefix option you can specify what's the prefix used in Tail input plugin, for the configuration example above the new configuration will look as follows:

So the proper for Kube_Tag_Prefix value must be composed by Tag prefix set in Tail input plugin plus the converted monitored directory replacing slashes with dots.

in normal operation mode allows to be configurable through or using specific arguments in the command line, while this is the ideal deployment case, there are scenarios where a more restricted configuration is required: static configuration mode.

Static configuration mode aims to include a built-in configuration in the final binary of Fluent Bit, disabling the usage of external files or flags at runtime.

Getting Started

Requirements

The following steps assumes you are familiar with configuring Fluent Bit using text files and you have experience building it from scratch as described in the section.

Configuration Directory

In your file system prepare a specific directory that will be used as an entry point for the build system to lookup and parse the configuration files. It is mandatory that this directory contain as a minimum one configuration file called fluent-bit.conf containing the required , and sections. As an example create a new fluent-bit.conf file with the following content:

the configuration provided above will calculate CPU metrics from the running system and print them to the standard output interface.

Build with Custom Configuration

Inside Fluent Bit source code, get into the build/ directory and run CMake appending the FLB_STATIC_CONF option pointing the configuration directory recently created, e.g:

then build it:

At this point the fluent-bit binary generated is ready to run without necessity of further configuration:

You can download the most recent stable or development source code.

Stable

For production systems, it's strongly suggested that you get the latest stable release of the source code in either zip file or tarball file format from GitHub using the following link pattern:

For example, for version 1.8.12 the link is:

Development

If you want to contribute to Fluent Bit, you should use the most recent code. You can get the development version from the Git repository:

The master branch is where the development of Fluent Bit happens. Development version users should expect issues when compiling or at run time.

Fluent Bit users are encouraged to help test every development version to ensure a stable release.

Fluent Bit is distributed as fluent-bit package and is available for long-term support releases of Ubuntu. The latest officially supported version is Noble Numbat (24.04).

Single line install

A simple installation script is provided to be used for most Linux targets. This will always install the most recent version released.

This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to follow the instructions below.

Server GPG key

The first step is to add our server GPG key to your keyring to ensure you can get our signed packages. Follow the official Debian wiki guidance:

Updated key from March 2022

From the 1.9.0 and 1.8.15 releases please note that the GPG key has been updated at so ensure this new one is added.

The GPG Key fingerprint of the new key is:

The previous key is still available at and may be required to install previous versions.

The GPG Key fingerprint of the old key is:

Update your sources lists

Update your repositories database

Now let your system update the apt database:

Install Fluent Bit

Using the following apt-get command you are able now to install the latest fluent-bit:

Now the following step is to instruct systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/syslog file.

Fluent Bit is distributed as fluent-bit package and is available for the Raspberry, specifically for distribution, the following versions are supported:

• Raspbian Bullseye (11)

• Raspbian Buster (10)

Server GPG key

The first step is to add our server GPG key to your keyring, on that way you can get our signed packages:

Updated key from March 2022

From the 1.9.0 and 1.8.15 releases please note that the GPG key has been updated at so ensure this new one is added.

The GPG Key fingerprint of the new key is:

The previous key is still available at and may be required to install previous versions.

The GPG Key fingerprint of the old key is:

Refer to the to see which platforms are supported in each release.

Update your sources lists

On Debian and derivative systems such as Raspbian, you need to add our APT server entry to your sources lists, please add the following content at bottom of your /etc/apt/sources.list file.

Raspbian 11 (Bullseye)

Raspbian 10 (Buster)

Update your repositories database

Now let your system update the apt database:

Install Fluent Bit

Using the following apt-get command you are able now to install the latest fluent-bit:

Now the following step is to instruct systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/syslog file.

Learn how to install Fluent Bit and the AWS output plugins on Amazon Linux 2 using .

Fluent Bit container images are available on Docker Hub ready for production usage. Current available images can be deployed in multiple architectures.

Quick Start

Get started by simply typing the following command:

Tags and Versions

The following table describes the Linux container tags that are available on Docker Hub repository:

It is strongly suggested that you always use the latest image of Fluent Bit.

Windows container images are provided from v2.0.6 for Windows Server 2019 and Windows Server 2022. These can be found as tags on the same Docker Hub registry above.

Multi Architecture Images

From a deployment perspective, there is no need to specify an architecture, the container client tool that pulls the image gets the proper layer for the running architecture.

Verify signed container images

Note: replace cosign above with the binary installed if it has a different name (e.g. cosign-linux-amd64).

Keyless signing is also provided but this is still experimental:

Getting Started

Download the last stable image from 2.0 series:

Once the image is in place, now run the following (useless) test which makes Fluent Bit measure CPU usage by the container:

That command will let Fluent Bit measure CPU usage every second and flush the results to the standard output, e.g:

F.A.Q

Why there is no Fluent Bit Docker image based on Alpine Linux ?

Alpine Linux uses Musl C library instead of Glibc. Musl is not fully compatible with Glibc which generated many issues in the following areas when used with Fluent Bit:

• Memory Allocator: to run Fluent Bit properly in high-load environments, we use Jemalloc as a default memory allocator which reduce fragmentation and provides better performance for our needs. Jemalloc cannot run smoothly with Musl and requires extra work.

• Alpine Linux Musl functions bootstrap have a compatibility issue when loading Golang shared libraries, this generate problems when trying to load Golang output plugins in Fluent Bit.

• Alpine Linux Musl Time format parser does not support Glibc extensions

• Maintainers preference in terms of base image due to security and maintenance reasons are Distroless and Debian.

Why use distroless containers ?

• Only include what you need, reduce the attack surface available.

• Reduces size so improves perfomance as well.

• Reduces false positives on scans (and reduces resources required for scanning).

• Reduces supply chain security requirements to just what you need.

• Helps prevent unauthorised processes or users interacting with the container.

• Less need to harden the container (and container runtime, K8S, etc.).

• Faster CICD processes.

With any choice of course there are downsides:

• No shell or package manager to update/add things.

  • Generally though dynamic updating is a bad idea in containers as the time it is done affects the outcome: two containers started at different times using the same base image may perform differently or get different dependencies, etc.

  • A better approach is to rebuild a new image version but then you can do this with Distroless, however it is harder requiring multistage builds or similar to provide the new dependencies.

• Debugging can be harder.

  • More specifically you need applications set up to properly expose information for debugging rather than rely on traditional debug approaches of connecting to processes or dumping memory. This can be an upfront cost vs a runtime cost but does shift left in the development process so hopefully is a reduction overall.

• Assumption that Distroless is secure: nothing is secure (just more or less secure) and there are still exploits so it does not remove the need for securing your system.

• Sometimes you need to use a common base image, e.g. with audit/security/health/etc. hooks integrated, or common base tooling (this could still be Distroless though).

One other important thing to note is that exec'ing into a container will potentially impact resource limits.

• This can be a quite different container from the one you want to investigate (e.g. lots of extra tools or even a different base).

• No resource limits applied to this container - can be good or bad.

• Runs in pod namespaces, just another container that can access everything the others can.

• May need architecture of the pod to share volumes, etc.

• Requires more recent versions of K8S and the container runtime plus RBAC allowing it.

AWS maintains a distribution of Fluent Bit that combines the latest official release with a set of Go Plugins for sending logs to AWS services. AWS and Fluent Bit are working together to rewrite their plugins for inclusion in the official Fluent Bit distribution.

Plugins

The image contains Go Plugins for:

• Amazon CloudWatch as cloudwatch_logs. See the or the .

• Amazon Kinesis Data Firehose as kinesis_firehose. See the or the .

• Amazon Kinesis Data Streams as kinesis_streams. See the or the .

These plugins are higher performance than Go plugins.

Also, Fluent Bit includes an S3 output plugin named s3.

Versions and Regional Repositories

AWS vends their container image using , and a set of highly available regional Amazon ECR repositories. For more information, see the .

The AWS for Fluent Bit image uses a custom versioning scheme because it contains multiple projects. To see what each release contains, see the .

SSM Public Parameters

AWS vends SSM public parameters with the regional repository link for each image. These parameters can be queried by any AWS account.

To see a list of available version tags in a given region, run the following command:

To see the ECR repository URI for a given image tag in a given region, run the following:

You can use these SSM public parameters as parameters in your CloudFormation templates:

is a lightweight and extensible Log Processor that comes with full support for Kubernetes:

• Process Kubernetes containers logs from the file system or Systemd/Journald.

• Enrich logs with Kubernetes Metadata.

• Centralize your logs in third party storage services like Elasticsearch, InfluxDB, HTTP, etc.

Concepts

Before getting started it is important to understand how Fluent Bit will be deployed. Kubernetes manages a cluster of nodes, so our log agent tool will need to run on every node to collect logs from every POD, hence Fluent Bit is deployed as a DaemonSet (a POD that runs on every node of the cluster).

When Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich each entry with the following information (metadata):

• Pod Name

• Pod ID

• Container Name

• Container ID

• Labels

• Annotations

To obtain this information, a built-in filter plugin called kubernetes talks to the Kubernetes API Server to retrieve relevant information such as the pod_id, labels and annotations, other fields such as pod_name, container_id and container_name are retrieved locally from the log file names. All of this is handled automatically, no intervention is required from a configuration aspect.

Installation

Note for OpenShift

If you are using Red Hat OpenShift you will also need to set up security context constraints (SCC) using the relevant option in the helm chart.

Installing with Helm Chart

To add the Fluent Helm Charts repo use the following command

To validate that the repo was added you can run helm search repo fluent to ensure the charts were added. The default chart can then be installed by running the following

Default Values

Details

The default configuration of Fluent Bit makes sure of the following:

• Consume all containers logs from the running Node and parse them with either the docker or cri multiline parser.

• Persist how far it got into each file it is tailing so if a pod is restarted it picks up from where it left off.

• The Kubernetes filter will enrich the logs with Kubernetes metadata, specifically labels and annotations. The filter only goes to the API Server when it cannot find the cached info, otherwise it uses the cache.

• There is an option called Retry_Limit set to False, that means if Fluent Bit cannot flush the records to Elasticsearch it will re-try indefinitely until it succeed.

Windows Deployment

Since v1.5.0, Fluent Bit supports deployment to Windows pods.

Log files overview

When deploying Fluent Bit to Kubernetes, there are three log files that you need to pay attention to.

C:\k\kubelet.err.log

• This is the error log file from kubelet daemon running on host.

• You will need to retain this file for future troubleshooting (to debug deployment failures etc.)

C:\var\log\containers\<pod>_<namespace>_<container>-<docker>.log

• This is the main log file you need to watch. Configure Fluent Bit to follow this file.

• It is actually a symlink to the Docker log file in C:\ProgramData\, with some additional metadata on its file name.

C:\ProgramData\Docker\containers\<docker>\<docker>.log

• This is the log file produced by Docker.

• Normally you don't directly read from this file, but you need to make sure that this file is visible from Fluent Bit.

Typically, your deployment yaml contains the following volume configuration.

Configure Fluent Bit

Mitigate unstable network on Windows pods

• DNS_Retries - Retries N times until the network start working (6)

• DNS_Wait_Time - Lookup interval between network status checks (30)

By default, Fluent Bit waits for 3 minutes (30 seconds x 6 times). If it's not enough for you, tweak the configuration as follows.

source code provides Bitbake recipes to configure, build and package the software for a Yocto based image. Note that specific steps of usage of these recipes in your Yocto environment (Poky) is out of the scope of this documentation.

We distribute two main recipes, one for testing/dev purposes and other with the latest stable release.

It's strongly recommended to always use the stable release of Fluent Bit recipe and not the one from GIT master for production deployments.

Fluent Bit and other architectures

Fluent Bit >= v1.1.x fully supports x86_64, x86, arm32v7 and arm64v8.

Before You Get Started

Fluent Bit traditionally offered a classic configuration mode, a custom configuration format that we are gradually phasing out. While classic mode has served well for many years, it has several limitations. Its basic design only supports grouping sections with key-value pairs and lacks the ability to handle sub-sections or complex data structures like lists.

YAML, now a mainstream configuration format, has become essential in a cloud ecosystem where everything is configured this way. To minimize friction and provide a more intuitive experience for creating data pipelines, we strongly encourage users to transition to YAML. The YAML format enables features, such as processors, that are not possible to configure in classic mode.

As of Fluent Bit v3.2, you can configure everything in YAML.

List of Available Sections

Configuring Fluent Bit with YAML introduces the following root-level sections:

Section Documentation

To access detailed configuration guides for each section, use the following links:

• • Overview of global settings, configuration options, and examples.

• • Detailed guide on defining parsers and supported formats.

• • Explanation of multiline parsing configuration.

• • Details on setting up pipelines and using processors.

• • How to load external plugins.

• • Guide on setting up and using upstream nodes with supported plugins.

• • Information on setting environment variables and their scope within Fluent Bit.

• • Description on how to include external YAML files.

Install on Redhat / CentOS

Fluent Bit is distributed as fluent-bit package and is available for the latest stable CentOS system.

The following architectures are supported

• x86_64

• aarch64 / arm64v8

For CentOS 9+ we use CentOS Stream as the canonical base system.

Single line install

A simple installation script is provided to be used for most Linux targets. This will always install the most recent version released.

This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to follow the instructions below.

CentOS 8

CentOS 8 is now EOL so the default Yum repositories are unavailable.

Make sure to configure to use an appropriate mirror, for example:

An alternative is to use Rocky or Alma Linux which should be equivalent.

Configure Yum

We provide fluent-bit through a Yum repository. In order to add the repository reference to your system, please add a new file called fluent-bit.repo in /etc/yum.repos.d/ with the following content:

It is best practice to always enable the gpgcheck and repo_gpgcheck for security reasons. We sign our repository metadata as well as all of our packages.

Updated key from March 2022

The GPG Key fingerprint of the new key is:

The GPG Key fingerprint of the old key is:

Install

Once your repository is configured, run the following command to install it:

Now the following step is to instruct Systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/messages file.

FAQ

Yum install fails with a "404 - Page not found" error for the package mirror

The fluent-bit.repo file for the latest installations of Fluent-Bit uses a $releasever variable to determine the correct version of the package to install to your system:

Depending on your Red Hat distribution version, this variable may return a value other than the OS major release version (e.g., RHEL7 Server distributions return "7Server" instead of just "7"). The Fluent-Bit package url uses just the major OS release version, so any other value here will cause a 404.

In order to resolve this issue, you can replace the $releasever variable with your system's OS major release version. For example:

Currently, Fluent Bit supports two configuration formats:

• : standard configuration format as of v3.2.

• : to be deprecated at the end of 2025.

Command line interface

Fluent Bit exposes most of it features through the command line interface. Running the -h option you can get a list of the options available:

Parsers enable Fluent Bit components to transform unstructured data into a structured internal representation. You can define parsers either directly in the main configuration file or in separate external files for better organization.

This page provides a general overview of how to declare parsers.

The main section name is parsers, and it allows you to define a list of parser configurations. The following example demonstrates how to set up two simple parsers:

You can define multiple parsers sections, either within the main configuration file or distributed across included files.

For more detailed information on parser options and advanced configurations, please refer to the section.

Fluent Bit is distributed as fluent-bit package and is available for the latest (and legacy) stable Debian systems: Bookworm and Bullseye. The following architectures are supported

• x86_64

• aarch64 / arm64v8

Single line install

A simple installation script is provided to be used for most Linux targets. This will always install the most recent version released.

This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to follow the instructions below.

Server GPG key

The first step is to add our server GPG key to your keyring, on that way you can get our signed packages. Follow the official Debian wiki guidance:

Updated key from March 2022

From the 1.9.0 and 1.8.15 releases please note that the GPG key has been updated at so ensure this new one is added.

The GPG Key fingerprint of the new key is:

The GPG Key fingerprint of the old key is:

Update your sources lists

Update your repositories database

Now let your system update the apt database:

Install Fluent Bit

Using the following apt-get command you are able now to install the latest fluent-bit:

Now the following step is to instruct systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/syslog file.

The service section defines global properties of the service. The available configuration keys are:

Configuration Example

Below is a simple configuration example that defines a service section and a pipeline with a random input and stdout output:

While Fluent Bit comes with a variety of built-in plugins, it also supports loading external plugins at runtime. This feature is especially useful for loading Go or Wasm plugins that are built as shared object files (.so). Fluent Bit's YAML configuration provides two ways to load these external plugins:

1. Inline YAML Section

You can specify external plugins directly within your main YAML configuration file using the plugins section. Here’s an example:

2. YAML Plugins File Included via plugins_file Option

Alternatively, you can load external plugins from a separate YAML file by specifying the plugins_file option in the service section. Here’s how to configure this:

In this setup, the extra_plugins.yaml file might contain the following plugins section:

Key Points

• Built-in vs. External: Fluent Bit comes with many built-in plugins, but you can load external plugins at runtime to extend the tool’s functionality.

• Loading Mechanism: External plugins must be shared object files (.so). You can define them inline in the main YAML configuration or include them from a separate YAML file for better modularity.

The env section allows you to define environment variables directly within the configuration file. These variables can then be used to dynamically replace values throughout your configuration using the ${VARIABLE_NAME} syntax.

Values set in the env section are case-sensitive. However, as a best practice, we recommend using uppercase names for environment variables. The example below defines two variables, FLUSH_INTERVAL and STDOUT_FMT, which can be accessed in the configuration using ${FLUSH_INTERVAL} and ${STDOUT_FMT}:

Predefined Variables

Fluent Bit provides a set of predefined environment variables that can be used in your configuration:

External Variables

In addition to variables defined in the configuration file or the predefined ones, Fluent Bit can access system environment variables set in the user space. These external variables can be referenced in the configuration using the same ${VARIABLE_NAME} pattern.

For example, to set the FLUSH_INTERVAL system environment variable to 2 and use it in your configuration:

In the configuration file, you can then access this value as follows:

This approach allows you to easily manage and override configuration values using environment variables, providing flexibility in various deployment environments.

The Upstream Servers section defines a group of endpoints, referred to as nodes, which are used by output plugins to distribute data in a round-robin fashion. This is particularly useful for plugins that require load balancing when sending data. Examples of plugins that support this capability include and .

In YAML, this section is named upstream_servers and requires specifying a name for the group and a list of nodes. Below is an example that defines two upstream server groups: forward-balancing and forward-balancing-2:

Key Concepts

• Nodes: Each node in the upstream_servers group must specify a name, host, and port. Additional settings like tls, tls_verify, and shared_key can be configured as needed for secure communication.

Usage Note

While the upstream_servers section can be defined globally, some output plugins may require the configuration to be specified in a separate YAML file. Be sure to consult the documentation for each specific output plugin to understand its requirements.

For more details, refer to the documentation of the respective output plugins.

Multiline parsers are used to combine logs that span multiple events into a single, cohesive message. This is particularly useful for handling stack traces, error logs, or any log entry that contains multiple lines of information.

In YAML configuration, the syntax for defining multiline parsers differs slightly from the classic configuration format introducing minor breaking changes, specifically on how the rules are defined.

Below is an example demonstrating how to define a multiline parser directly in the main configuration file, as well as how to include additional definitions from external files:

The example above defines a multiline parser named multiline-regex-test that uses regular expressions to handle multi-event logs. The parser contains two rules: the first rule transitions from start_state to cont when a matching log entry is detected, and the second rule continues to match subsequent lines.

For more detailed information on configuring multiline parsers, including advanced options and use cases, please refer to the Configuring Multiline Parsers section.

One of the ways to configure Fluent Bit is using a main configuration file. Fluent Bit allows the use one configuration file that works at a global scope and uses the defined .

The main configuration file supports four sections:

• Service

• Input

• Filter

• Output

It's also possible to split the main configuration file into multiple files using the Include File feature to include external files.

Service

The Service section defines global properties of the service. The following keys are:

The following is an example of a SERVICE section:

Config input

Name is mandatory and tells Fluent Bit which input plugin to load. Tag is mandatory for all plugins except for the input forward plugin, which provides dynamic tags.

Example

The following is an example of an INPUT section:

Config filter

The FILTER section defines a filter (related to an filter plugin). Each filter plugin can add it own configuration keys. The base configuration for each FILTER section contains:

Name is mandatory and lets Fluent Bit know which filter plugin should be loaded. Match or Match_Regex is mandatory for all plugins. If both are specified, Match_Regex takes precedence.

Filter example

The following is an example of a FILTER section:

Config output

The OUTPUT section specifies a destination that certain records should go to after a Tag match. Fluent Bit can route up to 256 OUTPUT plugins. The configuration supports the following keys:

Output example

The following is an example of an OUTPUT section:

Example: collecting CPU metrics

The following configuration file example demonstrates how to collect CPU metrics and flush the results every five seconds to the standard output:

Config Include File

To avoid complicated long configuration files is better to split specific parts in different files and call them (include) from one main file. The @INCLUDE can be used in the following way:

The configuration reader will try to open the path somefile.conf. If not found, the reader assumes the file is on a relative path based on the path of the base configuration file:

• Main configuration path: /tmp/main.conf

• Included file: somefile.conf

• Fluent Bit will try to open somefile.conf, if it fails it will try /tmp/somefile.conf.

The @INCLUDE command only works at top-left level of the configuration line, and can't be used inside sections.

Wildcard character (*) supports including multiple files. For example:

Files matching the wildcard character are included unsorted. If plugin ordering between files needs to be preserved, the files should be included explicitly.

Fluent Bit supports the usage of environment variables in any value associated to a key when using a configuration file.

The variables are case sensitive and can be used in the following format:

When Fluent Bit starts, the configuration reader will detect any request for ${MY_VARIABLE} and will try to resolve its value.

When Fluent Bit is running under systemd (using the official packages), environment variables can be set in the following files:

• /etc/default/fluent-bit (Debian based system)

• /etc/sysconfig/fluent-bit (Others)

These files are ignored if they do not exist.

Example

Create the following configuration file (fluent-bit.conf):

Open a terminal and set the environment variable:

The above command set the 'stdout' value to the variable MY_OUTPUT.

Run Fluent Bit with the recently created configuration file:

As you can see the service worked properly as the configuration was valid.

Fluent Bit is compatible with latest Apple macOS system on x86_64 and Apple Silicon architectures.

Installation Packages

The packages can be found here:

Requirements

For the next steps, you will need to have installed in your system. If is not there, you can install it with the following command:

Installing from Homebrew

The Fluent Bit package on Homebrew is not officially supported, but should work for basic use cases and testing. It can be installed using:

Compile from Source

Install build dependencies

Run the following brew command in your terminal to retrieve the dependencies:

Get the source and build it

Grab a fresh copy of the Fluent Bit source code (upstream):

Optionally, if you want to use a specific version, just checkout to the proper tag. If you want to use v1.8.13 just do:

In order to prepare the build system, we need to expose certain environment variables so Fluent Bit CMake build rules can pick the right libraries:

Change to the build/ directory inside the Fluent Bit sources:

Build Fluent Bit. Note that we are indicating to the build system "where" the final binaries and config files should be installed:

Install Fluent Bit to the directory specified above. Note that this requires root privileges due to the directory we will write information to:

The binaries and configuration examples can be located at /opt/fluent-bit/.

Create macOS installer from source

Grab a fresh copy of the Fluent Bit source code (upstream):

Optionally, if you want to use a specific version, just checkout to the proper tag. If you want to use v1.9.2 just do:

In order to prepare the build system, we need to expose certain environment variables so Fluent Bit CMake build rules can pick the right libraries:

And then, creating the specific macOS SDK target (For example, specifying macOS Big Sur (11.3) SDK environment):

Change to the build/ directory inside the Fluent Bit sources:

Build the Fluent Bit macOS installer.

Then, macOS installer will be generated as:

Finally, fluent-bit-<fluent-bit version>-(intel or apple).pkg will be generated.

The created installer will put binaries at /opt/fluent-bit/.

Running Fluent Bit

To make the access path easier to Fluent Bit binary, in your terminal extend the PATH variable:

Now as a simple test, try Fluent Bit by generating a simple dummy message which will be printed to the standard output interface every 1 second:

You will see an output similar to this:

To halt the process, press ctrl-c in the terminal.

Configuration files must be flexible enough for any deployment need, but they must keep a clean and readable format.

Fluent Bit Commands extends a configuration file with specific built-in features. The list of commands available as of Fluent Bit 0.12 series are:

@INCLUDE Command

Configuring a logging pipeline might lead to an extensive configuration file. In order to maintain a human-readable configuration, it's suggested to split the configuration in multiple files.

The @INCLUDE command allows the configuration reader to include an external configuration file, e.g:

The above example defines the main service configuration file and also include two files to continue the configuration:

inputs.conf

outputs.conf

Note that despites the order of inclusion, Fluent Bit will ALWAYS respect the following order:

• Service

• Inputs

• Filters

• Outputs

@SET Command

The @SET command can only be used at root level of each line, meaning it cannot be used inside a section, e.g:

Fluent Bit works internally with structured records and it can be composed of an unlimited number of keys and values. Values can be anything like a number, string, array, or a map.

Having a way to select a specific part of the record is critical for certain core functionalities or plugins, this feature is called Record Accessor.

consider Record Accessor a simple grammar to specify record content and other miscellaneous values.

Format

A record accessor rule starts with the character $. Using the structured content above as an example the following table describes how to access a record:

The following table describe some accessing rules and the expected returned value:

If the accessor key does not exist in the record like the last example $labels['undefined'] , the operation is simply omitted, no exception will occur.

Usage Example

The file content to process in test.log is the following:

Running Fluent Bit with the configuration above the output will be:

Limitations of record_accessor templating

The Fluent Bit record_accessor library has a limitation in the characters that can separate template variables- only dots and commas (. and ,) can come after a template variable. This is because the templating library must parse the template and determine the end of a variable.

The following would be invalid templates because the two template variables are not separated by commas or dots:

• $TaskID-$ECSContainerName

• $TaskID/$ECSContainerName

• $TaskID_$ECSContainerName

• $TaskIDfooo$ECSContainerName

However, the following are valid:

• $TaskID.$ECSContainerName

• $TaskID.ecs_resource.$ECSContainerName

• $TaskID.fooo.$ECSContainerName

And the following are valid since they only contain one template variable with nothing after it:

• fooo$TaskID

• fooo____$TaskID

• fooo/bar$TaskID

The pipeline section defines the flow of how data is collected, processed, and sent to its final destination. It encompasses the following core concepts:

Example Configuration

Here’s a simple example of a pipeline configuration:

Pipeline Processors

Processors operate on specific signals such as logs, metrics, and traces. They are attached to an input plugin and must specify the signal type they will process.

Example of a Processor

In the example below, the content_modifier processor inserts or updates (upserts) the key my_new_key with the value 123 for all log records generated by the tail plugin. This processor is only applied to log signals:

Here is a more complete example with multiple processors:

You might noticed that processors not only can be attached to input, but also to an output.

How Are Processors Different from Filters?

While processors and filters are similar in that they can transform, enrich, or drop data from the pipeline, there is a significant difference in how they operate:

• Processors: Run in the same thread as the input plugin when the input plugin is configured to be threaded (threaded: true). This design provides better performance, especially in multi-threaded setups.

• Filters: Run in the main event loop. When multiple filters are used, they can introduce performance overhead, particularly under heavy workloads.

Running Filters as Processors

Example of a Filter Running as a Processor

In the example below, the grep filter is used as a processor to filter log events based on a pattern:

The includes section allows you to specify additional YAML configuration files to be merged into the current configuration. These files are identified as a list of filenames and can include relative or absolute paths. If no absolute path is provided, the file is assumed to be located in a directory relative to the file that references it.

This feature is useful for organizing complex configurations into smaller, manageable files and including them as needed.

Usage

Below is an example demonstrating how to include additional YAML files using relative path references. This is the file system path structure

The content of fluent-bit.yaml

Key Points

• Relative Paths: If a path is not specified as absolute, it will be treated as relative to the file that includes it.

• Organized Configurations: Using the includes section helps keep your configuration modular and easier to maintain.

note: Ensure that the included files are formatted correctly and contain valid YAML configurations for seamless integration.

Fluent Bit might optionally use a configuration file to define how the service will behave.

Before proceeding we need to understand how the configuration schema works.

The schema is defined by three concepts:

• Sections

• Entries: Key/Value

• Indented Configuration Mode

A simple example of a configuration file is as follows:

Sections

A section is defined by a name or title inside brackets. Looking at the example above, a Service section has been set using [SERVICE] definition. Section rules:

• All section content must be indented (4 spaces ideally).

• Multiple sections can exist on the same file.

• A section is expected to have comments and entries, it cannot be empty.

• Any commented line under a section, must be indented too.

• End-of-line comments are not supported, only full-line comments.

Entries: Key/Value

A section may contain Entries, an entry is defined by a line of text that contains a Key and a Value, using the above example, the [SERVICE] section contains two entries, one is the key Daemon with value off and the other is the key Log_Level with the value debug. Entries rules:

• An entry is defined by a key and a value.

• A key must be indented.

• A key must contain a value which ends in the breakline.

• Multiple keys with the same name can exist.

Also commented lines are set prefixing the # character, those lines are not processed but they must be indented too.

Indented Configuration Mode

Fluent Bit configuration files are based in a strict Indented Mode, that means that each configuration file must follow the same pattern of alignment from left to right when writing text. By default an indentation level of four spaces from left to right is suggested. Example:

As you can see there are two sections with multiple entries and comments, note also that empty lines are allowed and they do not need to be indented.

has an engine that helps to coordinate the data ingestion from input plugins. The engine calls the scheduler to decide when it's time to flush the data through one or multiple output plugins. The scheduler flushes new data at a fixed number of seconds, and retries when asked.

When an output plugin gets called to flush some data, after processing that data it can notify the engine using these possible return statuses:

• OK: Data successfully processed and flushed.

• Retry: If a retry is requested, the engine asks the scheduler to retry flushing that data. The scheduler decides how many seconds to wait before retry.

• Error: An unrecoverable error occurred and the engine shouldn't try to flush that data again.

Configure wait time for retry

The scheduler provides two configuration options, called scheduler.cap and scheduler.base, which can be set in the Service section. These determine the waiting time before a retry happens.

The scheduler.base determines the lower bound of time and the scheduler.cap determines the upper bound for each retry.

Fluent Bit uses an exponential backoff and jitter algorithm to determine the waiting time before a retry. The waiting time is a random number between a configurable upper and lower bound. For a detailed explanation of the exponential backoff and jitter algorithm, see .

For example:

For the Nth retry, the lower bound of the random number will be:

base

The upper bound will be:

min(base * (Nth power of 2), cap)

For example:

When base is set to 3 and cap is set to 30:

First retry: The lower bound will be 3. The upper bound will be 3 * 2 = 6. The waiting time will be a random number between (3, 6).

Second retry: The lower bound will be 3. The upper bound will be 3 * (2 * 2) = 12. The waiting time will be a random number between (3, 12).

Third retry: The lower bound will be 3. The upper bound will be 3 * (2 * 2 * 2) =24. The waiting time will be a random number between (3, 24).

Fourth retry: The lower bound will be 3, because 3 * (2 * 2 * 2 * 2) = 48 > 30. The upper bound will be 30. The waiting time will be a random number between (3, 30).

Wait time example

The following example configures the scheduler.base as 3 seconds and scheduler.cap as 30 seconds.

The waiting time will be:

Configure retries

The scheduler provides a configuration option called Retry_Limit, which can be set independently for each output section. This option lets you disable retries or impose a limit to try N times and then discard the data after reaching that limit:

Retry example

The following example configures two outputs, where the HTTP plugin has an unlimited number of retries, and the Elasticsearch plugin have a limit of 5 retries:

Certain configuration directives in Fluent Bit refer to unit sizes such as when defining the size of a buffer or specific limits, we can find these in plugins like , or in generic properties like .

Starting from v0.11.10, all unit sizes have been standardized across the core and plugins, the following table describes the options that can be used and what they mean:

Install Fluent Bit in your embedded Linux system.

Install

To install, select Fluent Bit in your defconfig. See the Config.in file for all configuration options.

Run

The default configuration file is written to:

Fluent Bit is started by the S99fluent-bit script.

Support

All configurations with a toolchain that supports threads and dynamic library linking are supported.

Fluent Bit is distributed as fluent-bit package for Windows and as a . Fluent Bit has two flavours of Windows installers: a ZIP archive (for quick testing) and an EXE installer (for system installation).

Not all plugins are supported on Windows: the shows the default set of supported plugins.

Configuration

Make sure to provide a valid Windows configuration with the installation, a sample one is shown below:

Migration to Fluent Bit

From version 1.9, td-agent-bit is a deprecated package and was removed after 1.9.9. The correct package name to use now is fluent-bit.

Installation Packages

The latest stable version is 3.2.3. Each version is available via the following download URLs.

Note these are now using the Github Actions built versions, the legacy AppVeyor builds are still available (AMD 32/64 only) at releases.fluentbit.io but are deprecated.

MSI installers are also available:

To check the integrity, use Get-FileHash cmdlet on PowerShell.

Installing from ZIP archive

Download a ZIP archive from above. There are installers for 32-bit and 64-bit environments, so choose one suitable for your environment.

Then you need to expand the ZIP archive. You can do this by clicking "Extract All" on Explorer, or if you're using PowerShell, you can use Expand-Archive cmdlet.

The ZIP package contains the following set of files.

Now, launch cmd.exe or PowerShell on your machine, and execute fluent-bit.exe as follows.

If you see the following output, it's working fine!

To halt the process, press CTRL-C in the terminal.

Installing from EXE installer

Download an EXE installer from above. It has both 32-bit and 64-bit builds. Choose one which is suitable for you.

Double-click the EXE installer you've downloaded. The installation wizard will automatically start.

Click Next and proceed. By default, Fluent Bit is installed into C:\Program Files\fluent-bit\, so you should be able to launch fluent-bit as follows after installation.

Installer options

To silently install to C:\fluent-bit directory here is an example:

The uninstaller automatically provided also supports a silent un-install using the same /S flag. This may be useful for provisioning with automation like Ansible, Puppet, etc.

Windows Service Support

Windows services are equivalent to "daemons" in UNIX (i.e. long-running background processes). Since v1.5.0, Fluent Bit has the native support for Windows Service.

Suppose you have the following installation layout:

To register Fluent Bit as a Windows service, you need to execute the following command on Command Prompt. Please be careful that a single space is required after binpath=.

Now Fluent Bit can be started and managed as a normal Windows service.

To halt the Fluent Bit service, just execute the "stop" command.

To start Fluent Bit automatically on boot, execute the following:

[FAQ] Fluent Bit fails to start up when installed under C:\Program Files

Quotations are required if file paths contain spaces. Here is an example:

[FAQ] How can I manage Fluent Bit service via PowerShell?

Instead of sc.exe, PowerShell can be used to manage Windows services.

Create a Fluent Bit service:

Start the service:

Query the service status:

Stop the service:

Remove the service (requires PowerShell 6.0 or later)

Compile from Source

If you need to create a custom executable, you can use the following procedure to compile Fluent Bit by yourself.

Preparation

First, you need Microsoft Visual C++ to compile Fluent Bit. You can install the minimum toolkit by the following command:

When asked which packages to install, choose "C++ Build Tools" (make sure that "C++ CMake tools for Windows" is selected too) and wait until the process finishes.

It is important to have installed OpenSSL binaries, at least the library files and headers.

Compilation

Open the start menu on Windows and type "Command Prompt for VS". From the result list select the one that corresponds to your target system ( x86 or x64).

Note: Check that the installed OpenSSL library files match the selected target. You can check the library files by using the dumpbin command with the /headers option .

Clone the source code of Fluent Bit.

Compile the source code.

Now you should be able to run Fluent Bit:

Packaging

To create a ZIP package, call cpack as follows: