• Event or Record

• Filtering

• Tag

• Timestamp

• Match

• Structured Message

Event or Record

Every incoming piece of data that belongs to a log or a metric that is retrieved by Fluent Bit is considered an Event or a Record.

As an example consider the following content of a Syslog file:

Jan 18 12:52:16 flb systemd[2222]: Starting GNOME Terminal Server
Jan 18 12:52:16 flb dbus-daemon[2243]: [session uid=1000 pid=2243] Successfully activated service 'org.gnome.Terminal'
Jan 18 12:52:16 flb systemd[2222]: Started GNOME Terminal Server.
Jan 18 12:52:16 flb gsd-media-keys[2640]: # watch_fast: "/org/gnome/terminal/legacy/" (establishing: 0, active: 0)

It contains four lines and all of them represents four independent Events.

Internally an Event is comprised of:

• timestamp

• key/value metadata (since v2.1.0)

• payload

Event format

The Fluent Bit wire protocol represents an Event as a 2-element array with a nested array as the first element:

[[TIMESTAMP, METADATA], MESSAGE]

where

• TIMESTAMP is a timestamp in seconds as an integer or floating point value (not a string);

• METADATA is a possibly-empty object containing event metadata; and

• MESSAGE is an object containing the event body.

Fluent Bit versions prior to v2.1.0 instead used:

[TIMESTAMP, MESSAGE]

to represent events. This format is still supported for reading input event streams.

Filtering

In some cases it is required to perform modifications on the Events content, the process to alter, enrich or drop Events is called Filtering.

There are many use cases when Filtering is required like:

• Append specific information to the Event like an IP address or metadata.

• Select a specific piece of the Event content.

• Drop Events that matches certain pattern.

Tag

Every Event that gets into Fluent Bit gets assigned a Tag. This tag is an internal string that is used in a later stage by the Router to decide which Filter or Output phase it must go through.

Most of the tags are assigned manually in the configuration. If a tag is not specified, Fluent Bit will assign the name of the Input plugin instance from where that Event was generated from.

Timestamp

The Timestamp represents the time when an Event was created. Every Event contains a Timestamp associated. The Timestamp is a numeric fractional integer in the format:

SECONDS.NANOSECONDS

Seconds

It is the number of seconds that have elapsed since the Unix epoch.

Nanoseconds

Fractional second or one thousand-millionth of a second.

Match

Fluent Bit allows to deliver your collected and processed Events to one or multiple destinations, this is done through a routing phase. A Match represent a simple rule to select Events where it Tags matches a defined rule.

Structured Messages

Source events can have or not have a structure. A structure defines a set of keys and values inside the Event message. As an example consider the following two messages:

No structured message

"Project Fluent Bit created on 1398289291"

Structured Message

{"project": "Fluent Bit", "created": 1398289291}

At a low level both are just an array of bytes, but the Structured message defines keys and values, having a structure helps to implement faster operations on data modifications.

Dealing with raw strings or unstructured messages is a constant pain; having a structure is highly desired. Ideally we want to set a structure to the incoming data by the Input Plugins as soon as they are collected:

The Parser allows you to convert from unstructured to structured data. As a demonstrative example consider the following Apache (HTTP Server) log entry:

192.168.2.20 - - [28/Jul/2006:10:27:10 -0300] "GET /cgi-bin/try/ HTTP/1.0" 200 3395

The above log line is a raw string without format, ideally we would like to give it a structure that can be processed later easily. If the proper configuration is used, the log entry could be converted to:

{
  "host":    "192.168.2.20",
  "user":    "-",
  "method":  "GET",
  "path":    "/cgi-bin/try/",
  "code":    "200",
  "size":    "3395",
  "referer": "",
  "agent":   ""
 }

Rather than serving as a drop-in replacement, Fluent Bit enhances the observability strategy for your infrastructure by adapting and optimizing your existing logging layer, as well as metrics and traces processing. Furthermore, Fluent Bit supports a vendor-neutral approach, seamlessly integrating with other ecosystems such as Prometheus and OpenTelemetry. Trusted by major cloud providers, banks, and companies in need of a ready-to-use telemetry agent solution, Fluent Bit effectively manages diverse data sources and formats while maintaining optimal performance.

Fluent Bit can be deployed as an edge agent for localized telemetry data handling or utilized as a central aggregator/collector for managing telemetry data across multiple sources and environments.

The output interface allows us to define destinations for the data. Common destinations are remote services, local file system or standard interface with others. Outputs are implemented as plugins and there are many available.

When an output plugin is loaded, an internal instance is created. Every instance has its own independent configuration. Configuration keys are often called properties.

Every output plugin has its own documentation section specifying how it can be used and what properties are available.

• Licensed under the terms of Apache License v2.0

• Production Grade solutions: deployed million of times every single day.

• Vendor neutral and community driven projects

• Widely Adopted by the Industry: trusted by all major companies like AWS, Microsoft, Google Cloud and hundreds of others.

The following table describes a comparison of different areas of the projects:

When an input plugin is loaded, an internal instance is created. Every instance has its own and independent configuration. Configuration keys are often called properties.

Every input plugin has its own documentation section where it's specified how it can be used and what properties are available.

After the project was around for some time, it got more traction for normal Linux systems, also with the new containerized world, the Cloud Native community asked to extend the project scope to support more sources, filters, and destinations. Not so long after, Fluent Bit became one of the preferred solutions to solve the logging challenges in Cloud environments.

Fluent Bit as buffering strategies go, offers a primary buffering mechanism in memory and an optional secondary one using the file system. With this hybrid solution you can accommodate any use case safely and keep a high performance while processing your data.

Both mechanisms are not mutually exclusive and when the data is ready to be processed or delivered it will always be in memory, while other data in the queue might be in the file system until is ready to be processed and moved up to memory.

Features

• High Performance: High throughput with low resources consumption

• Data Parsing
• Metrics Support: Prometheus and OpenTelemetry compatible

• Reliability and Data Integrity
• Networking

  • Security: built-in TLS/SSL support

  • Asynchronous I/O

• • More than 100 built-in plugins are available

  • Extensibility

    • Write any input, filter or output plugin in C language
• • Create new streams of data using query results

  • Aggregation Windows

  • Data analysis and prediction: Timeseries forecasting

• Portable: runs on Linux, macOS, Windows and BSD systems

Fluent Bit, Fluentd and CNCF

In production environments we want to have full control of the data we are collecting, filtering is an important feature that allows us to alter the data before delivering it to some destination.

Filtering is implemented through plugins, so each filter available could be used to match, exclude or enrich your logs with some specific metadata.

We support many filters, A common use case for filtering is Kubernetes deployments. Every Pod log needs to get the proper metadata associated

Very similar to the input plugins, Filters run in an instance context, which has its own independent configuration. Configuration keys are often called properties.

The following article cover the relevant notes for users upgrading from previous Fluent Bit versions. We aim to cover compatibility changes that you must be aware of.

Note: release notes will be prepared in advance of a Git tag for a release so an official release should provide both a tag and a release note together to allow users to verify and understand the release contents.

The tag drives the overall binary release process so release binaries (containers/packages) will appear after a tag and its associated release note. This allows users to expect the new release binary to appear and allow/deny/update it as appropriate in their infrastructure.

Fluent Bit v1.9.9

The td-agent-bit package is no longer provided after this release. Users should switch to the fluent-bit package.

Fluent Bit v1.6

If you are migrating from previous version of Fluent Bit please review the following important changes:

Tail Input Plugin

Now by default the plugin follows a file from the end once the service starts (old behavior was always read from the beginning). For every file found at start, its followed from it last position, for new files discovered at runtime or rotated, they are read from the beginning.

If you desire to keep the old behavior you can set the option read_from_head to true.

Stackdriver Output Plugin

If you have any existing queries based on the resource's project_id, please update your query accordingly.

Fluent Bit v1.5

The migration from v1.4 to v1.5 is pretty straightforward.

Fluent Bit v1.4

If you are migrating from Fluent Bit v1.3, there are no breaking changes. Just new exciting features to enjoy :)

Fluent Bit v1.3

If you are migrating from Fluent Bit v1.2 to v1.3, there are no breaking changes. If you are upgrading from an older version please review the incremental changes below.

Fluent Bit v1.2

Docker, JSON, Parsers and Decoders

On Fluent Bit v1.2 we have fixed many issues associated with JSON encoding and decoding, for hence when parsing Docker logs is no longer necessary to use decoders. The new Docker parser looks like this:

[PARSER]
    Name         docker
    Format       json
    Time_Key     time
    Time_Format  %Y-%m-%dT%H:%M:%S.%L
    Time_Keep    On

Note: again, do not use decoders.

Kubernetes Filter

We have done improvements also on how Kubernetes Filter handle the stringified log message. If the option Merge_Log is enabled, it will try to handle the log content as a JSON map, if so, it will add the keys to the root map.

In addition, we have fixed and improved the option called Merge_Log_Key. If a merge log succeed, all new keys will be packaged under the key specified by this option, a suggested configuration is as follows:

[FILTER]
    Name             Kubernetes
    Match            kube.*
    Kube_Tag_Prefix  kube.var.log.containers.
    Merge_Log        On
    Merge_Log_Key    log_processed

As an example, if the original log content is the following map:

{"key1": "val1", "key2": "val2"}

the final record will be composed as follows:

{
    "log": "{\"key1\": \"val1\", \"key2\": \"val2\"}",
    "log_processed": {
        "key1": "val1",
        "key2": "val2"
    }
}

Fluent Bit v1.1

If you are upgrading from Fluent Bit <= 1.0.x you should take in consideration the following relevant changes when switching to Fluent Bit v1.1 series:

Kubernetes Filter

We introduced a new configuration property called Kube_Tag_Prefix to help Tag prefix resolution and address an unexpected behavior that landed in previous versions.

During 1.0.x release cycle, a commit in Tail input plugin changed the default behavior on how the Tag was composed when using the wildcard for expansion generating breaking compatibility with other services. Consider the following configuration example:

[INPUT]
    Name  tail
    Path  /var/log/containers/*.log
    Tag   kube.*

The expected behavior is that Tag will be expanded to:

kube.var.log.containers.apache.log

but the change introduced in 1.0 series switched from absolute path to the base file name only:

kube.apache.log

On Fluent Bit v1.1 release we restored to our default behavior and now the Tag is composed using the absolute path of the monitored file.

Having absolute path in the Tag is relevant for routing and flexible configuration where it also helps to keep compatibility with Fluentd behavior.

This behavior switch in Tail input plugin affects how Filter Kubernetes operates. As you know when the filter is used it needs to perform local metadata lookup that comes from the file names when using Tail as a source. Now with the new Kube_Tag_Prefix option you can specify what's the prefix used in Tail input plugin, for the configuration example above the new configuration will look as follows:

[INPUT]
    Name  tail
    Path  /var/log/containers/*.log
    Tag   kube.*

[FILTER]
    Name             kubernetes
    Match            *
    Kube_Tag_Prefix  kube.var.log.containers.

So the proper for Kube_Tag_Prefix value must be composed by Tag prefix set in Tail input plugin plus the converted monitored directory replacing slashes with dots.

There are two important concepts in Routing:

• Tag

• Match

When the data is generated by the input plugins, it comes with a Tag (most of the time the Tag is configured manually), the Tag is a human-readable indicator that helps to identify the data source.

In order to define where the data should be routed, a Match rule must be specified in the output configuration.

Consider the following configuration example that aims to deliver CPU metrics to an Elasticsearch database and Memory metrics to the standard output interface:

[INPUT]
    Name cpu
    Tag  my_cpu

[INPUT]
    Name mem
    Tag  my_mem

[OUTPUT]
    Name   es
    Match  my_cpu

[OUTPUT]
    Name   stdout
    Match  my_mem

Note: the above is a simple example demonstrating how Routing is configured.

Routing works automatically reading the Input Tags and the Output Match rules. If some data has a Tag that doesn't match upon routing time, the data is deleted.

Routing with Wildcard

Routing is flexible enough to support wildcard in the Match pattern. The below example defines a common destination for both sources of data:

[INPUT]
    Name cpu
    Tag  my_cpu

[INPUT]
    Name mem
    Tag  my_mem

[OUTPUT]
    Name   stdout
    Match  my_*

The match rule is set to my_* which means it will match any Tag that starts with my_.

Container Deployment

Install on Linux (Packages)

Install on Windows (Packages)

Install on macOS (Packages)

Compile from Source (Linux, Windows, FreeBSD, macOS)

Sandbox Environment

If you are interested in learning about Fluent Bit you can try out the sandbox environment

Enterprise Packages

The buffer phase already contains the data in an immutable state, meaning, no other filter can be applied.

Fluent Bit offers a buffering mechanism in the file system that acts as a backup system to avoid data loss in case of system failures.

• Compiler: GCC or clang

• CMake

• Flex & Bison: only if you enable the Stream Processor or Record Accessor feature (both enabled by default)

• Libyaml development headers and libraries

In the core there are not other dependencies, For certain features that depends on third party components like output plugins with special backend libraries (e.g: kafka), those are included in the main source code repository.

The following operating systems and architectures are supported in Fluent Bit.

From an architecture support perspective, Fluent Bit is fully functional on x86_64, Arm64v8 and Arm32v7 based processors.

Stable

For production systems, we strongly suggest that you always get the latest stable release of the source code in either zip or tarball format from Github using the following link pattern:

https://github.com/fluent/fluent-bit/archive/refs/tags/v<release version>.tar.gz https://github.com/fluent/fluent-bit/archive/refs/tags/v<release version>.zip

Development

For anyone who aims to contribute to the project by testing or extending the code base, you can get the development version from our GIT repository:

$ git clone https://github.com/fluent/fluent-bit

Note that our master branch is where the development of Fluent Bit happens. Since it's a development version, expect issues when compiling or at run time.

We encourage everybody to help us testing every development version, at the end this is what will become stable.

Fluent Bit is distributed as fluent-bit package and is available for the latest (and legacy) stable Debian systems: Bookworm and Bullseye. The following architectures are supported

• x86_64

• aarch64 / arm64v8

Single line install

A simple installation script is provided to be used for most Linux targets. This will always install the most recent version released.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to follow the instructions below.

Server GPG key

curl https://packages.fluentbit.io/fluentbit.key | gpg --dearmor > /usr/share/keyrings/fluentbit-keyring.gpg

Updated key from March 2022

The GPG Key fingerprint of the new key is:

C3C0 A285 34B9 293E AF51  FABD 9F9D DC08 3888 C1CD
Fluentbit releases (Releases signing key) <releases@fluentbit.io>

The GPG Key fingerprint of the old key is:

F209 D876 2A60 CD49 E680 633B 4FF8 368B 6EA0 722A

Update your sources lists

deb [signed-by=/usr/share/keyrings/fluentbit-keyring.gpg] https://packages.fluentbit.io/debian/${CODENAME} ${CODENAME} main

Update your repositories database

Now let your system update the apt database:

sudo apt-get update

Install Fluent Bit

Using the following apt-get command you are able now to install the latest fluent-bit:

sudo apt-get install fluent-bit

Now the following step is to instruct systemd to enable the service:

sudo systemctl start fluent-bit

If you do a status check, you should see a similar output like this:

sudo service fluent-bit status
● fluent-bit.service - Fluent Bit
   Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
   Active: active (running) since mié 2016-07-06 16:58:25 CST; 2h 45min ago
 Main PID: 6739 (fluent-bit)
    Tasks: 1
   Memory: 656.0K
      CPU: 1.393s
   CGroup: /system.slice/fluent-bit.service
           └─6739 /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf
...

The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/syslog file.

Static configuration mode aims to include a built-in configuration in the final binary of Fluent Bit, disabling the usage of external files or flags at runtime.

Getting Started

Requirements

Configuration Directory

[SERVICE]
    Flush     1
    Daemon    off
    Log_Level info

[INPUT]
    Name      cpu

[OUTPUT]
    Name      stdout
    Match     *

the configuration provided above will calculate CPU metrics from the running system and print them to the standard output interface.

Build with Custom Configuration

Inside Fluent Bit source code, get into the build/ directory and run CMake appending the FLB_STATIC_CONF option pointing the configuration directory recently created, e.g:

$ cd fluent-bit/build/
$ cmake -DFLB_STATIC_CONF=/path/to/my/confdir/

then build it:

$ make

At this point the fluent-bit binary generated is ready to run without necessity of further configuration:

$ bin/fluent-bit 
Fluent-Bit v0.15.0
Copyright (C) Treasure Data

[2018/10/19 15:32:31] [ info] [engine] started (pid=15186)
[0] cpu.local: [1539984752.000347547, {"cpu_p"=>0.750000, "user_p"=>0.500000, "system_p"=>0.250000, "cpu0.p_cpu"=>1.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>0.000000, "cpu1.p_cpu"=>0.000000, "cpu1.p_user"=>0.000000, "cpu1.p_system"=>0.000000, "cpu2.p_cpu"=>0.000000, "cpu2.p_user"=>0.000000, "cpu2.p_system"=>0.000000, "cpu3.p_cpu"=>1.000000, "cpu3.p_user"=>1.000000, "cpu3.p_system"=>0.000000}]

The most secure option is to create the repositories acccording to the instructions for your specific OS.

A simple installation script is provided to be used for most Linux targets. This will by default install the most recent version released.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

This is purely a convenience helper and should always be validated prior to use.

GPG key updates

The GPG Key fingerprint of the new key is:

C3C0 A285 34B9 293E AF51  FABD 9F9D DC08 3888 C1CD
Fluentbit releases (Releases signing key) <releases@fluentbit.io>

The GPG Key fingerprint of the old key is:

F209 D876 2A60 CD49 E680 633B 4FF8 368B 6EA0 722A

Migration to Fluent Bit

From version 1.9, td-agent-bit is a deprecated package and is removed after 1.9.9. The correct package name to use now is fluent-bit.

AWS maintains a distribution of Fluent Bit combining the latest official release with a set of Go Plugins for sending logs to AWS services. AWS and Fluent Bit are working together to rewrite their plugins for inclusion in the official Fluent Bit distribution.

Plugins

Fluent Bit includes Amazon CloudWatch Logs plugin named cloudwatch_logs, Amazon Kinesis Firehose plugin named kinesis_firehose and Amazon Kinesis Data Streams plugin named kinesis_streams which are higher performance than Go plugins.

Also, Fluent Bit includes S3 output plugin named s3.

Versions and Regional Repositories

SSM Public Parameters

AWS vends SSM Public Parameters with the regional repository link for each image. These parameters can be queried by any AWS account.

To see a list of available version tags in a given region, run the following command:

To see the ECR repository URI for a given image tag in a given region, run the following:

You can use these SSM public parameters as parameters in your CloudFormation templates:

• Process Kubernetes containers logs from the file system or Systemd/Journald.

• Enrich logs with Kubernetes Metadata.

• Centralize your logs in third party storage services like Elasticsearch, InfluxDB, HTTP, etc.

Concepts

Before getting started it is important to understand how Fluent Bit will be deployed. Kubernetes manages a cluster of nodes, so our log agent tool will need to run on every node to collect logs from every POD, hence Fluent Bit is deployed as a DaemonSet (a POD that runs on every node of the cluster).

When Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich each entry with the following information (metadata):

• Pod Name

• Pod ID

• Container Name

• Container ID

• Labels

• Annotations

To obtain this information, a built-in filter plugin called kubernetes talks to the Kubernetes API Server to retrieve relevant information such as the pod_id, labels and annotations, other fields such as pod_name, container_id and container_name are retrieved locally from the log file names. All of this is handled automatically, no intervention is required from a configuration aspect.

Installation

The recommended way to deploy Fluent Bit is with the official Helm Chart: https://github.com/fluent/helm-charts

Note for OpenShift

If you are using Red Hat OpenShift you will also need to set up security context constraints (SCC):

Installing with Helm Chart

To add the Fluent Helm Charts repo use the following command

To validate that the repo was added you can run helm search repo fluent to ensure the charts were added. The default chart can then be installed by running the following

Default Values

Details

The default configuration of Fluent Bit makes sure of the following:

• Consume all containers logs from the running Node.

• The Kubernetes filter will enrich the logs with Kubernetes metadata, specifically labels and annotations. The filter only goes to the API Server when it cannot find the cached info, otherwise it uses the cache.

• There is an option called Retry_Limit set to False, that means if Fluent Bit cannot flush the records to Elasticsearch it will re-try indefinitely until it succeed.

Container Runtime Interface (CRI) parser

Fluent Bit by default assumes that logs are formatted by the Docker interface standard. However, when using CRI you can run into issues with malformed JSON if you do not modify the parser used. Fluent Bit includes a CRI log parser that can be used instead. An example of the parser is seen below:

To use this parser change the Input section for your configuration from docker to cri

Windows Deployment

Since v1.5.0, Fluent Bit supports deployment to Windows pods.

Log files overview

When deploying Fluent Bit to Kubernetes, there are three log files that you need to pay attention to.

C:\k\kubelet.err.log

• This is the error log file from kubelet daemon running on host.

• You will need to retain this file for future troubleshooting (to debug deployment failures etc.)

C:\var\log\containers\<pod>_<namespace>_<container>-<docker>.log

• This is the main log file you need to watch. Configure Fluent Bit to follow this file.

• It is actually a symlink to the Docker log file in C:\ProgramData\, with some additional metadata on its file name.

C:\ProgramData\Docker\containers\<docker>\<docker>.log

• This is the log file produced by Docker.

• Normally you don't directly read from this file, but you need to make sure that this file is visible from Fluent Bit.

Typically, your deployment yaml contains the following volume configuration.

Configure Fluent Bit

Mitigate unstable network on Windows pods

• DNS_Retries - Retries N times until the network start working (6)

• DNS_Wait_Time - Lookup interval between network status checks (30)

By default, Fluent Bit waits for 3 minutes (30 seconds x 6 times). If it's not enough for you, tweak the configuration as follows.

Install on Amazon Linux

Fluent Bit is distributed as fluent-bit package and is available for the latest Amazon Linux 2 and Amazon Linux 2023. The following architectures are supported

• x86_64

• aarch64 / arm64v8

Single line install

A simple installation script is provided to be used for most Linux targets. This will always install the most recent version released.

This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to follow the instructions below.

Amazon Linux 2022

Amazon Linux 2022 was previously supported but is removed since it became GA Amazon Linux 2023

Configure Yum

We provide fluent-bit through a Yum repository. In order to add the repository reference to your system, please add a new file called fluent-bit.repo in /etc/yum.repos.d/ with the following content:

Amazon Linux 2

Amazon Linux 2023

Note: we encourage you always enable the gpgcheck for security reasons. All our packages are signed.

Updated key from March 2022

The GPG Key fingerprint of the new key is:

The GPG Key fingerprint of the old key is:

Install

Once your repository is configured, run the following command to install it:

Now the following step is to instruct systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/messages file.

Install on Redhat / CentOS

Fluent Bit is distributed as fluent-bit package and is available for the latest stable CentOS system.

The following architectures are supported

• x86_64

• aarch64 / arm64v8

For CentOS 9+ we use CentOS Stream as the canonical base system.

Single line install

A simple installation script is provided to be used for most Linux targets. This will always install the most recent version released.

This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to follow the instructions below.

CentOS 8

CentOS 8 is now EOL so the default Yum repositories are unavailable.

Make sure to configure to use an appropriate mirror, for example:

An alternative is to use Rocky or Alma Linux which should be equivalent.

Configure Yum

We provide fluent-bit through a Yum repository. In order to add the repository reference to your system, please add a new file called fluent-bit.repo in /etc/yum.repos.d/ with the following content:

It is best practice to always enable the gpgcheck and repo_gpgcheck for security reasons. We sign our repository metadata as well as all of our packages.

Updated key from March 2022

The GPG Key fingerprint of the new key is:

The GPG Key fingerprint of the old key is:

Install

Once your repository is configured, run the following command to install it:

Now the following step is to instruct Systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/messages file.

FAQ

Yum install fails with a "404 - Page not found" error for the package mirror

The fluent-bit.repo file for the latest installations of Fluent-Bit uses a $releasever variable to determine the correct version of the package to install to your system:

Depending on your Red Hat distribution version, this variable may return a value other than the OS major release version (e.g., RHEL7 Server distributions return "7Server" instead of just "7"). The Fluent-Bit package url uses just the major OS release version, so any other value here will cause a 404.

In order to resolve this issue, you can replace the $releasever variable with your system's OS major release version. For example:

Requirements

• CMake >= 3.12

• Flex

• Bison >= 3

• YAML headers

• OpenSSL headers

Prepare environment

In the following steps you can find exact commands to build and install the project with the default options. If you already know how CMake works you can skip this part and look at the build options available. Note that Fluent Bit requires CMake 3.x. You may need to use cmake3 instead of cmake to complete the following steps on your system.

Change to the build/ directory inside the Fluent Bit sources:

Now you are ready to start the compilation process through the simple make command:

to continue installing the binary on the system just do:

it's likely you may need root privileges so you can try to prefixing the command with sudo.

Build Options

Fluent Bit provides certain options to CMake that can be enabled or disabled when configuring, please refer to the following tables under the General Options, Development Options, Input Plugins and _Output Plugins sections.

General Options

Development Options

Optimization Options

Input Plugins

The input plugins provides certain features to gather information from a specific source type which can be a network interface, some built-in metric or through a specific input device, the following input plugins are available:

Filter Plugins

The filter plugins allows to modify, enrich or drop records. The following table describes the filters available on this version:

Output Plugins

The output plugins gives the capacity to flush the information to some external interface, service or terminal, the following table describes the output plugins available as of this version:

• Raspbian Bullseye (11)

• Raspbian Buster (10)

Server GPG key

The first step is to add our server GPG key to your keyring, on that way you can get our signed packages:

curl https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -

Updated key from March 2022

The GPG Key fingerprint of the new key is:

C3C0 A285 34B9 293E AF51  FABD 9F9D DC08 3888 C1CD
Fluentbit releases (Releases signing key) <releases@fluentbit.io>

The GPG Key fingerprint of the old key is:

F209 D876 2A60 CD49 E680 633B 4FF8 368B 6EA0 722A

Update your sources lists

On Debian and derivative systems such as Raspbian, you need to add our APT server entry to your sources lists, please add the following content at bottom of your /etc/apt/sources.list file.

Raspbian 11 (Bullseye)

deb https://packages.fluentbit.io/raspbian/bullseye bullseye main

Raspbian 10 (Buster)

deb https://packages.fluentbit.io/raspbian/buster buster main

Update your repositories database

Now let your system update the apt database:

sudo apt-get update

Install Fluent Bit

Using the following apt-get command you are able now to install the latest fluent-bit:

sudo apt-get install fluent-bit

Now the following step is to instruct systemd to enable the service:

sudo service fluent-bit start

If you do a status check, you should see a similar output like this:

sudo service fluent-bit status
● fluent-bit.service - Fluent Bit
   Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
   Active: active (running) since mié 2016-07-06 16:58:25 CST; 2h 45min ago
 Main PID: 6739 (fluent-bit)
    Tasks: 1
   Memory: 656.0K
      CPU: 1.393s
   CGroup: /system.slice/fluent-bit.service
           └─6739 /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf
...

The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/syslog file.

We distribute two main recipes, one for testing/dev purposes and other with the latest stable release.

It's strongly recommended to always use the stable release of Fluent Bit recipe and not the one from GIT master for production deployments.

Fluent Bit and other architectures

Fluent Bit >= v1.1.x fully supports x86_64, x86, arm32v7 and arm64v8.

Fluent Bit is distributed as fluent-bit package and is available for the latest stable Ubuntu system: Jammy Jellyfish.

Single line install

A simple installation script is provided to be used for most Linux targets. This will always install the most recent version released.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to follow the instructions below.

Server GPG key

curl https://packages.fluentbit.io/fluentbit.key | gpg --dearmor > /usr/share/keyrings/fluentbit-keyring.gpg

Updated key from March 2022

The GPG Key fingerprint of the new key is:

C3C0 A285 34B9 293E AF51  FABD 9F9D DC08 3888 C1CD
Fluentbit releases (Releases signing key) <releases@fluentbit.io>

The GPG Key fingerprint of the old key is:

F209 D876 2A60 CD49 E680 633B 4FF8 368B 6EA0 722A

Update your sources lists

deb [signed-by=/usr/share/keyrings/fluentbit-keyring.gpg] https://packages.fluentbit.io/ubuntu/${CODENAME} ${CODENAME} main

Update your repositories database

Now let your system update the apt database:

sudo apt-get update

Install Fluent Bit

Using the following apt-get command you are able now to install the latest fluent-bit:

sudo apt-get install fluent-bit

Now the following step is to instruct systemd to enable the service:

sudo systemctl start fluent-bit

If you do a status check, you should see a similar output like this:

systemctl status fluent-bit
● fluent-bit.service - Fluent Bit
   Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
   Active: active (running) since mié 2016-07-06 16:58:25 CST; 2h 45min ago
 Main PID: 6739 (fluent-bit)
    Tasks: 1
   Memory: 656.0K
      CPU: 1.393s
   CGroup: /system.slice/fluent-bit.service
           └─6739 /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf
...

The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/syslog file.

Fluent Bit container images are available on Docker Hub ready for production usage. Current available images can be deployed in multiple architectures.

Quick Start

Get started by simply typing the following command:

docker run -ti cr.fluentbit.io/fluent/fluent-bit

Tags and Versions

It is strongly suggested that you always use the latest image of Fluent Bit.

Windows container images are provided from v2.0.6 for Windows Server 2019 and Windows Server 2022. These can be found as tags on the same Docker Hub registry above.

Multi Architecture Images

From a deployment perspective, there is no need to specify an architecture, the container client tool that pulls the image gets the proper layer for the running architecture.

Verify signed container images

$ cosign verify --key "https://packages.fluentbit.io/fluentbit-cosign.pub" fluent/fluent-bit:2.0.6

Verification for index.docker.io/fluent/fluent-bit:2.0.6 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

[{"critical":{"identity":{"docker-reference":"index.docker.io/fluent/fluent-bit"},"image":{"docker-manifest-digest":"sha256:c740f90b07f42823d4ecf4d5e168f32ffb4b8bcd87bc41df8f5e3d14e8272903"},"type":"cosign container image signature"},"optional":{"release":"2.0.6","repo":"fluent/fluent-bit","workflow":"Release from staging"}}]

Note: replace cosign above with the binary installed if it has a different name (e.g. cosign-linux-amd64).

Keyless signing is also provided but this is still experimental:

COSIGN_EXPERIMENTAL=1 cosign verify fluent/fluent-bit:2.0.6

Getting Started

Download the last stable image from 2.0 series:

docker pull cr.fluentbit.io/fluent/fluent-bit:2.0

Once the image is in place, now run the following (useless) test which makes Fluent Bit measure CPU usage by the container:

docker run -ti cr.fluentbit.io/fluent/fluent-bit:2.0 \
  -i cpu -o stdout -f 1

That command will let Fluent Bit measure CPU usage every second and flush the results to the standard output, e.g:

[2019/10/01 12:29:02] [ info] [engine] started
[0] cpu.0: [1504290543.000487750, {"cpu_p"=>0.750000, "user_p"=>0.250000, "system_p"=>0.500000, "cpu0.p_cpu"=>0.000000, "cpu0.p_user"=>0.000000, "cpu0.p_system"=>0.000000, "cpu1.p_cpu"=>1.000000, "cpu1.p_user"=>0.000000, "cpu1.p_system"=>1.000000, "cpu2.p_cpu"=>1.000000, "cpu2.p_user"=>1.000000, "cpu2.p_system"=>0.000000, "cpu3.p_cpu"=>0.000000, "cpu3.p_user"=>0.000000, "cpu3.p_system"=>0.000000}]

F.A.Q

Why there is no Fluent Bit Docker image based on Alpine Linux ?

Alpine Linux uses Musl C library instead of Glibc. Musl is not fully compatible with Glibc which generated many issues in the following areas when used with Fluent Bit:

• Memory Allocator: to run Fluent Bit properly in high-load environments, we use Jemalloc as a default memory allocator which reduce fragmentation and provides better performance for our needs. Jemalloc cannot run smoothly with Musl and requires extra work.

• Alpine Linux Musl functions bootstrap have a compatibility issue when loading Golang shared libraries, this generate problems when trying to load Golang output plugins in Fluent Bit.

• Alpine Linux Musl Time format parser does not support Glibc extensions

• Maintainers preference in terms of base image due to security and maintenance reasons are Distroless and Debian.

Why use distroless containers ?

• Only include what you need, reduce the attack surface available.

• Reduces size so improves perfomance as well.

• Reduces false positives on scans (and reduces resources required for scanning).

• Reduces supply chain security requirements to just what you need.

• Helps prevent unauthorised processes or users interacting with the container.

• Less need to harden the container (and container runtime, K8S, etc.).

• Faster CICD processes.

With any choice of course there are downsides:

• No shell or package manager to update/add things.

  • Generally though dynamic updating is a bad idea in containers as the time it is done affects the outcome: two containers started at different times using the same base image may perform differently or get different dependencies, etc.

  • A better approach is to rebuild a new image version but then you can do this with Distroless, however it is harder requiring multistage builds or similar to provide the new dependencies.

• Debugging can be harder.

  • More specifically you need applications set up to properly expose information for debugging rather than rely on traditional debug approaches of connecting to processes or dumping memory. This can be an upfront cost vs a runtime cost but does shift left in the development process so hopefully is a reduction overall.

• Assumption that Distroless is secure: nothing is secure (just more or less secure) and there are still exploits so it does not remove the need for securing your system.

• Sometimes you need to use a common base image, e.g. with audit/security/health/etc. hooks integrated, or common base tooling (this could still be Distroless though).

One other important thing to note is that exec'ing into a container will potentially impact resource limits.

• This can be a quite different container from the one you want to investigate (e.g. lots of extra tools or even a different base).

• No resource limits applied to this container - can be good or bad.

• Runs in pod namespaces, just another container that can access everything the others can.

• May need architecture of the pod to share volumes, etc.

• Requires more recent versions of K8S and the container runtime plus RBAC allowing it.

Currently, Fluent Bit supports two configuration formats:

CLI flags

Fluent Bit also supports a CLI interface with various flags matching up to the configuration options available.

The main configuration file supports four types of sections:

• Service

• Input

• Filter

• Output

In addition, it's also possible to split the main configuration file in multiple files using the feature to include external files:

• Include File

Service

The Service section defines global properties of the service, the keys available as of this version are described in the following table:

The following is an example of a SERVICE section:

Input

An INPUT section defines a source (related to an input plugin), here we will describe the base configuration for each INPUT section. Note that each input plugin may add it own configuration keys:

The Name is mandatory and it let Fluent Bit know which input plugin should be loaded. The Tag is mandatory for all plugins except for the input forward plugin (as it provides dynamic tags).

Example

The following is an example of an INPUT section:

Filter

A FILTER section defines a filter (related to an filter plugin), here we will describe the base configuration for each FILTER section. Note that each filter plugin may add it own configuration keys:

The Name is mandatory and it let Fluent Bit know which filter plugin should be loaded. The Match or Match_Regex is mandatory for all plugins. If both are specified, Match_Regex takes precedence.

Example

The following is an example of an FILTER section:

Output

The OUTPUT section specify a destination that certain records should follow after a Tag match. Currently, Fluent Bit can route up to 256 OUTPUT plugins. The configuration support the following keys:

Example

The following is an example of an OUTPUT section:

Example: collecting CPU metrics

The following configuration file example demonstrates how to collect CPU metrics and flush the results every five seconds to the standard output:

Visualize

Include File

To avoid complicated long configuration files is better to split specific parts in different files and call them (include) from one main file.

Starting from Fluent Bit 0.12 the new configuration command @INCLUDE has been added and can be used in the following way:

The configuration reader will try to open the path somefile.conf, if not found, it will assume it's a relative path based on the path of the base configuration file, e.g:

• Main configuration file path: /tmp/main.conf

• Included file: somefile.conf

• Fluent Bit will try to open somefile.conf, if it fails it will try /tmp/somefile.conf.

The @INCLUDE command only works at top-left level of the configuration line, it cannot be used inside sections.

Wildcard character (*) is supported to include multiple files, e.g:

Note files matching the wildcard character are included unsorted. If plugins ordering between files need to be preserved, the files should be included explicitly.

An Upstream defines a set of nodes that will be targeted by an output plugin, by the nature of the implementation an output plugin must support the Upstream feature. The following plugin(s) have Upstream support:

The current balancing mode implemented is round-robin.

Configuration

To define an Upstream it's required to create an specific configuration file that contains an UPSTREAM and one or multiple NODE sections. The following table describe the properties associated to each section. Note that all of them are mandatory:

Nodes and specific plugin configuration

A Node might contain additional configuration keys required by the plugin, on that way we provide enough flexibility for the output plugin, a common use case is Forward output where if TLS is enabled, it requires a shared key (more details in the example below).

Nodes and TLS (Transport Layer Security)

In addition to the properties defined in the table above, the network operations against a defined node can optionally be done through the use of TLS for further encryption and certificates use.

Configuration File Example

The following example defines an Upstream called forward-balancing which aims to be used by Forward output plugin, it register three Nodes:

• node-1: connects to 127.0.0.1:43000

• node-2: connects to 127.0.0.1:44000

• node-3: connects to 127.0.0.1:45000 using TLS without verification. It also defines a specific configuration option required by Forward output called shared_key.