The following section will guide you to the step to download, build and install Fluent Bit from sources and specific instructions for the installation of binaries that we already distribute for Debian/Ubuntu/Redhat/CentOS and Raspberry Pi.

If you find some problem on a certain step, don't hesitate to report the problem on our bug tracker:

https://github.com/fluent/fluent-bit/issues

Fluent Bit is a Fast and Lightweight Log Processor and Forwarder for Linux, OSX and BSD family operating systems. It has been made with a strong focus on performance to allow the collection of events from different sources without complexity.

Fluent Bit is part of the Fluentd project ecosystem, it's licensed under the terms of the Apache License v2.0. This project is made and sponsored by Treasure Data.

Data collection matters and nowadays the scenarios from where the information can come from are very variable. For hence to be more flexible in certain markets needs, we may need different options. On this page, we will describe the relationship between the Fluentd and Fluent Bit open source projects.

Fluentd and Fluent Bit projects are both created and sponsored by Treasure Data and they aim to solve the collection, processing, and delivery of Logs.

Both projects share a lot of similarities, Fluent Bit is fully based on the design and experience of Fluentd architecture and general design. Choosing which one to use depends on the final needs, from an architecture perspective we can consider:

• Fluentd is a log collector, processor, and aggregator.

• Fluent Bit is a log collector and processor (it doesn't have strong aggregation features like Fluentd).

The following table describes a comparison in different areas of the projects:

Consider Fluentd mainly as an Aggregator and Fluent Bit as a Log Forwarder, we can see both projects complement each other providing a full reliable solution.

is an open source and multi-platform log forwarder tool which aims to be a generic Swiss knife for log collection and distribution.

We, , as a Big Data company, provide an analytics infrastructure in the Cloud where we provide an end-to-end solution to collect, store and do analytics over the data. is an integral part of this pipeline where it solves the log collection needs.

Being an open source project, it has been widely adopted to solve logging needs in Cloud Native environments where Docker and Kubernetes are key components; Fluent Bit is a natural fit.

Data collection and log forwarding is hard.

Nowadays the number of sources of information in our environments is ever increasing. Handling data collection at scale is complex, and collecting and aggregating diverse data requires a specialized tool that can deal with:

• Different sources of information.

• Different data formats.

• Multiple destinations.

was born to address the need for a high performance and optimized tool that can collect data from any input source, unify that data and deliver it to multiple destinations.

uses very low CPU and Memory consumption, it's compatible with most of x86, x86_64, AArch32 and AArch64 based platforms. In order to build it you need the following components in your system:

• Compiler: GCC or clang

• CMake

• Flex (only if Stream Processor is enabled)

• Bison (only if Stream Processor is enabled)

There are not other dependencies besides libc and pthreads in the most basic mode. For certain features that depends on third party components, those are included in the main source code repository.

The following operating systems and architectures are supported in Fluent Bit.

From an architecture support perspective, Fluent Bit is fully functional on x86, x86_64, AArch32 and AArch64 based processors.

Fluent Bit can work also on OSX and *BSD systems, but not all plugins will be available on all platforms. Official support will be expanding based on community demand.

Stable

For production systems, we strongly suggest that you always get the latest stable release from our web site, you can get the official tarballs (.tar.gz) from the following link:

Development

For people who aims to contribute to the project testing or extending the code base, can get the development version from our GIT repository:

Note that our master branch is where the development of Fluent Bit happens. Since it's a development version, expect issues when compiling or at run time.

We encourage everybody to help us testing every development version, at the end this is what will become stable.

The following article cover the relevant notes for users upgrading from previous Fluent Bit versions. We aim to cover compatibility changes that you must be aware of.

For more details about changes on each release please refer to the .

Fluent Bit v1.2

Docker, JSON, Parsers and Decoders

On Fluent Bit v1.2 we have fixed many issues associated with JSON encoding and decoding, for hence when parsing Docker logs is not longer necessary to use decoders. The new Docker parser looks like this:

Note: again, do not use decoders.

Kubernetes Filter

We have done improvements also on how Kubernetes Filter handle the stringified log message. If the option Merge_Log is enabled, it will try to handle the log content as a JSON map, if so, it will add the keys to the root map.

In addition, we have fixed and improved the option called Merge_Log_Key. If a merge log succeed, all new keys will be packaged under the key specified by this option, a suggested configuration is as follows:

As an example, if the original log content is the following map:

the final record will be composed as follows:

Fluent Bit v1.1

If you are upgrading from Fluent Bit <= 1.0.x you should take in consideration the following relevant changes when switching to Fluent Bit v1.1 series:

Kubernetes Filter

We introduced a new configuration property called Kube_Tag_Prefix to help Tag prefix resolution and address an unexpected behavior that landed in previous versions.

Duing 1.0.x release cycle, a commit in Tail input plugin changed the default behavior on how the Tag was composed when using the wildcard for expansion generating breaking compatibility with other services. Consider the following configuration example:

The expected behavior is that Tag will be expanded to:

but the change introduced in 1.0 series switched from absolute path to the base file name only:

On Fluent Bit v1.1 release we restored to our default behavior and now the Tag is composed using the absolute path of the monitored file.

Having absolute path in the Tag is relevant for routing and flexible configuration where it also helps to keep compatibility with Fluentd behavior.

This behavior switch in Tail input plugin affects how Filter Kubernetes operates. As you know when the filter is used it needs to perform local metadata lookup that comes from the file names when using Tail as a source. Now with the new Kube_Tag_Prefix option you can specify what's the prefix used in Tail input plugin, for the configuration example above the new configuration will look as follows:

So the proper for Kube_Tag_Prefix value must be composed by Tag prefix set in Tail input plugin plus the converted monitored directory replacing slashes with dots.

Fluent Bit is distributed as td-agent-bit package and is available for the latest stable Debian system: Jessie. This stable Fluent Bit distribution package is maintained by .

Server GPG key

The first step is to add our server GPG key to your keyring, on that way you can get our signed packages:

Update your sources lists

On Debian, you need to add our APT server entry to your sources lists, please add the following content at bottom of your /etc/apt/sources.list file:

Debian 9 (Stretch)

Debian 8 (Jessie)

Update your repositories database

Now let your system update the apt database:

Install TD-Agent Bit

Using the following apt-get command you are able now to install the latest td-agent-bit:

Now the following step is to instruct systemd to enable the service:

If you do a status check, you should see a similar output like this:

The default configuration of td-agent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/syslog file.

in normal operation mode allows to be configurable through or using specific arguments in the command line, while this is the ideal deployment case, there are scenarios where a more restricted configuration is required: static configuration mode.

Static configuration mode aims to include a built-in configuration in the final binary of Fluent Bit, disabling the usage of external files or flags at runtime.

Getting Started

Requirements

The following steps assumes you are familiar with configuring Fluent Bit using text files and you have experience building it from scratch as described in the section.

Configuration Directory

In your file system prepare a specific directory that will be used as an entry point for the build system to lookup and parse the configuration files. It is mandatory that this directory contain as a minimum one configuration file called fluent-bit.conf containing the required , and sections. As an example create a new fluent-bit.conf file with the following content:

the configuration provided above will calculate CPU metrics from the running system and print them to the standard output interface.

Build with Custom Configuration

Inside Fluent Bit source code, get into the build/ directory and run CMake appending the FLB_STATIC_CONF option pointing the configuration directory recently created, e.g:

then build it:

At this point the fluent-bit binary generated is ready to run without necessity of further configuration: