Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Every project has a story
In 2014, the Fluentd team at Treasure Data was forecasting the need for a lightweight log processor for constraint environments like embedded Linux and gateways. The project aimed to be part of the Fluentd ecosystem. At that moment, Eduardo Silva created Fluent Bit, a new open source solution, written from scratch and available under the terms of the Apache License v2.0.
After the project matured, it gained traction for normal Linux systems. With the new containerized world, the Cloud Native community asked to extend the project scope to support more sources, filters, and destinations. Not long after, Fluent Bit became one of the preferred solutions to solve the logging challenges in Cloud environments.
Learn these key concepts to understand how Fluent Bit operates.
Before diving into Fluent Bit you might want to get acquainted with some of the key concepts of the service. This document provides an introduction to those concepts and common Fluent Bit terminology. Reading this document will help you gain a more general understanding of the following topics:
Event or Record
Filtering
Tag
Timestamp
Match
Structured Message
Every incoming piece of data that belongs to a log or a metric that's retrieved by Fluent Bit is considered an Event or a Record.
As an example, consider the following content of a Syslog file:
It contains four lines that represent four independent Events.
An Event is comprised of:
timestamp
key/value metadata (v2.1.0 and greater)
payload
The Fluent Bit wire protocol represents an Event as a two-element array with a nested array as the first element:
where
TIMESTAMP
is a timestamp in seconds as an integer or floating point value (not a string).
METADATA
is an object containing event metadata, and might be empty.
MESSAGE
is an object containing the event body.
Fluent Bit versions prior to v2.1.0 used:
to represent events. This format is still supported for reading input event streams.
You might need to perform modifications on an Event's content. The process to alter, append to, or drop Events is called filtering.
Use filtering to:
Append specific information to the Event like an IP address or metadata.
Select a specific piece of the Event content.
Drop Events that match a certain pattern.
Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string used in a later stage by the Router to decide which Filter or Output phase it must go through.
Most tags are assigned manually in the configuration. If a tag isn't specified, Fluent Bit assigns the name of the Input plugin instance where that Event was generated from.
The Forward input plugin doesn't assign tags. This plugin speaks the Fluentd wire protocol called Forward where every Event already comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the client.
A tagged record must always have a Matching rule. To learn more about Tags and Matches, see Routing.
The timestamp represents the time an Event was created. Every Event contains an associated timestamps. All events have timestamps, and they're set by the input plugin or discovered through a data parsing process.
The timestamp is a numeric fractional integer in the format:
where:
_SECONDS_
is the number of seconds that have elapsed since the Unix epoch.
_NANOSECONDS_
is a fractional second or one thousand-millionth of a second.
Fluent Bit lets you route your collected and processed Events to one or multiple destinations. A Match represents a rule to select Events where a Tag matches a defined rule.
To learn more about Tags and Matches, see Routing.
Source events can have a structure. A structure defines a set of keys
and values
inside the Event message to implement faster operations on data modifications. Fluent Bit treats every Event message as a structured message.
Consider the following two messages:
No structured message
With a structured message
For performance reasons, Fluent Bit uses a binary serialization data format called MessagePack.
Fluent Bit license description
Fluent Bit, including its core, plugins, and tools are distributed under the terms of the Apache License v2.0:
High Performance Telemetry Agent for Logs, Metrics and Traces
Fluent Bit is a fast and lightweight telemetry agent for logs, metrics, and traces for Linux, macOS, Windows, and BSD family operating systems. Fluent Bit has been made with a strong focus on performance to allow the collection and processing of telemetry data from different sources without complexity.
High performance: High throughput with low resources consumption
Metrics support: Prometheus and OpenTelemetry compatible
Reliability and data integrity
Backpressure handling
Data buffering in memory and file system
Networking
Security: Built-in TLS/SSL support
Asynchronous I/O
Pluggable architecture and extensibility: Inputs, Filters and Outputs:
Connect nearly any source to nearly any destination using preexisting plugins
Extensibility:
Write input, filter, or output plugins in the C language
WASM: WASM Filter Plugins or WASM Input Plugins
Write Filters in Lua or Output plugins in Golang
Monitoring: Expose internal metrics over HTTP in JSON and Prometheus format
Stream Processing: Perform data selection and transformation using simple SQL queries
Create new streams of data using query results
Aggregation windows
Data analysis and prediction: Timeseries forecasting
Portable: Runs on Linux, macOS, Windows and BSD systems
Fluent Bit is a CNCF graduated sub-project under the umbrella of Fluentd. Fluent Bit is licensed under the terms of the Apache License v2.0.
Fluent Bit was originally created by Eduardo Silva and is now sponsored by Chronosphere. As a CNCF-hosted project, it is a fully vendor-neutral and community-driven project.
Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
Fluent Bit is an open source telemetry agent specifically designed to efficiently handle the challenges of collecting and processing telemetry data across a wide range of environments, from constrained systems to complex cloud infrastructures. Managing telemetry data from various sources and formats can be a constant challenge, particularly when performance is a critical factor.
Rather than serving as a drop-in replacement, Fluent Bit enhances the observability strategy for your infrastructure by adapting and optimizing your existing logging layer, and adding metrics and traces processing. Fluent Bit supports a vendor-neutral approach, seamlessly integrating with other ecosystems such as Prometheus and OpenTelemetry. Trusted by major cloud providers, banks, and companies in need of a ready-to-use telemetry agent solution, Fluent Bit effectively manages diverse data sources and formats while maintaining optimal performance and keeping resource consumption low.
Fluent Bit can be deployed as an edge agent for localized telemetry data handling or utilized as a central aggregator/collector for managing telemetry data across multiple sources and environments.
The production grade telemetry ecosystem
Telemetry data processing can be complex, especially at scale. That's why Fluentd was created. Fluentd is more than a simple tool, it's grown into a fullscale ecosystem that contains SDKs for different languages and subprojects like Fluent Bit.
Here, we describe the relationship between the Fluentd and Fluent Bit open source projects.
Both projects are:
Licensed under the terms of Apache License v2.0.
Graduated hosted projects by the Cloud Native Computing Foundation (CNCF).
Production grade solutions: Deployed millions of times every single day.
Vendor neutral and community driven.
Widely adopted by the industry: Trusted by major companies like AWS, Microsoft, Google Cloud, and hundreds of others.
The projects have many similarities: Fluent Bit is designed and built on top of the best ideas of Fluentd architecture and general design. Which one you choose depends on your end-users' needs.
The following table describes a comparison of different areas of the projects:
Attribute | Fluentd | Fluent Bit |
---|---|---|
Both Fluentd and Fluent Bit can work as Aggregators or Forwarders, and can complement each other or be used as standalone solutions.
In the recent years, cloud providers have switched from Fluentd to Fluent Bit for performance and compatibility. Fluent Bit is now considered the next-generation solution.
Performance and data safety
When processes data, it uses the system memory (heap) as a primary and temporary place to store the record logs before they get delivered. The records are processed in this private memory area.
Buffering is the ability to store the records, and continue storing incoming data while previous data is processed and delivered. Buffering in memory is the fastest mechanism, but there are scenarios requiring special strategies to deal with , data safety, or to reduce memory consumption by the service in constrained environments.
Network failures or latency in third party service is common. When data can't be delivered fast enough and new data to process arrives, the system can face backpressure.
Fluent Bit buffering strategies are designed to solve problems associated with backpressure and general delivery failures. Fluent Bit offers a primary buffering mechanism in memory and an optional secondary one using the file system. With this hybrid solution you can accommodate any use case safely and keep a high performance while processing your data.
These mechanisms aren't mutually exclusive. When data is ready to be processed or delivered it's always be in memory, while other data in the queue might be in the file system until is ready to be processed and moved up to memory.
To learn more about the buffering configuration in Fluent Bit, see .
Convert unstructured messages to structured messages
Dealing with raw strings or unstructured messages is difficult. Having a structure makes data more usable. Set a structure to the incoming data by using input plugins as data is collected:
The parser converts unstructured data to structured data. As an example, consider the following Apache (HTTP Server) log entry:
This log line is a raw string without format. Structuring the log makes it easier to process the data later. If the is used, the log entry could be converted to:
Parsers are fully configurable and are independently and optionally handled by each input plugin. For more details, see .
The way to gather data from your sources
provides input plugins to gather information from different sources. Some plugins collect data from log files, while others can gather metrics information from the operating system. There are many plugins to suit different needs.
When an input plugin loads, an internal instance is created. Each instance has its own independent configuration. Configuration keys are often called properties.
Every input plugin has its own documentation section that specifies how to use it and what properties are available.
For more details, see .
Data processing with reliability
The phase in the pipeline aims to provide a unified and persistent mechanism to store your data, using the primary in-memory model or the file system-based mode.
The buffer
phase contains the data in an immutable state, meaning that no other filter can be applied.
Buffered data uses the Fluent Bit internal binary representation, which isn't raw text.
Fluent Bit offers a buffering mechanism in the file system that acts as a backup system to avoid data loss in case of system failures.
Modify, enrich or drop your records
In production environments you need full control of the data you're collecting. Filtering lets you alter the collected data before delivering it to a destination.
Filtering is implemented through plugins. Each available filter can be used to match, exclude, or enrich your logs with specific metadata.
Fluent Bit support many filters. A common use case for filtering is Kubernetes deployments. Every pod log needs the proper metadata associated with it.
Like input plugins, filters run in an instance context, which has its own independent configuration. Configuration keys are often called properties.
For more details about the Filters available and their usage, see .