Before You Get Started

Fluent Bit traditionally offered a classic configuration mode, a custom configuration format that we are gradually phasing out. While classic mode has served well for many years, it has several limitations. Its basic design only supports grouping sections with key-value pairs and lacks the ability to handle sub-sections or complex data structures like lists.

YAML, now a mainstream configuration format, has become essential in a cloud ecosystem where everything is configured this way. To minimize friction and provide a more intuitive experience for creating data pipelines, we strongly encourage users to transition to YAML. The YAML format enables features, such as processors, that are not possible to configure in classic mode.

As of Fluent Bit v3.2, you can configure everything in YAML.

List of Available Sections

Configuring Fluent Bit with YAML introduces the following root-level sections:

Section Documentation

To access detailed configuration guides for each section, use the following links:

• Service Section documentation

  • Overview of global settings, configuration options, and examples.

• Parsers Section documentation

  • Detailed guide on defining parsers and supported formats.

• Multiline Parsers Section documentation

  • Explanation of multiline parsing configuration.

• Pipeline Section documentation

  • Details on setting up pipelines and using processors.

• Plugins Section documentation

  • How to load external plugins.

• Upstream Servers Section documentation

  • Guide on setting up and using upstream nodes with supported plugins.

• Environment Variables Section documentation

  • Information on setting environment variables and their scope within Fluent Bit.

• Includes Section documentation

  • Description on how to include external YAML files.

The service section defines global properties of the service. The available configuration keys are:

Configuration Example

Below is a simple configuration example that defines a service section with hot reloading enabled and a pipeline with a random input and stdout output:

service:
  flush: 1
  log_level: info
  http_server: true
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: random

  outputs:
    - name: stdout
      match: '*'

Parsers enable Fluent Bit components to transform unstructured data into a structured internal representation. You can define parsers either directly in the main configuration file or in separate external files for better organization.

This page provides a general overview of how to declare parsers.

The main section name is parsers, and it allows you to define a list of parser configurations. The following example demonstrates how to set up two simple parsers:

parsers:
  - name: json
    format: json

  - name: docker
    format: json
    time_key: time
    time_format: "%Y-%m-%dT%H:%M:%S.%L"
    time_keep: true

You can define multiple parsers sections, either within the main configuration file or distributed across included files.

For more detailed information on parser options and advanced configurations, please refer to the Configuring Parsers section.

Multiline parsers are used to combine logs that span multiple events into a single, cohesive message. This is particularly useful for handling stack traces, error logs, or any log entry that contains multiple lines of information.

In YAML configuration, the syntax for defining multiline parsers differs slightly from the classic configuration format introducing minor breaking changes, specifically on how the rules are defined.

Below is an example demonstrating how to define a multiline parser directly in the main configuration file, as well as how to include additional definitions from external files:

multiline_parsers:
  - name: multiline-regex-test
    type: regex
    flush_timeout: 1000
    rules:
      - state: start_state
        regex: '/([a-zA-Z]+ \d+ \d+:\d+:\d+)(.*)/'
        next_state: cont
      - state: cont
        regex: '/^\s+at.*/'
        next_state: cont

The example above defines a multiline parser named multiline-regex-test that uses regular expressions to handle multi-event logs. The parser contains two rules: the first rule transitions from start_state to cont when a matching log entry is detected, and the second rule continues to match subsequent lines.

For more detailed information on configuring multiline parsers, including advanced options and use cases, please refer to the Configuring Multiline Parsers section.

The pipeline section defines the flow of how data is collected, processed, and sent to its final destination. It encompasses the following core concepts:

Example Configuration

Here’s a simple example of a pipeline configuration:

pipeline:
  inputs:
    - name: tail
      path: /var/log/example.log
      parser: json

      processors:
        logs:
          - name: record_modifier
  filters:
    - name: grep
      match: '*'
      regex: key pattern

  outputs:
    - name: stdout
      match: '*'

Pipeline Processors

Processors operate on specific signals such as logs, metrics, and traces. They are attached to an input plugin and must specify the signal type they will process.

Example of a Processor

In the example below, the content_modifier processor inserts or updates (upserts) the key my_new_key with the value 123 for all log records generated by the tail plugin. This processor is only applied to log signals:

parsers:
  - name: json
    format: json

pipeline:
  inputs:
    - name: tail
      path: /var/log/example.log
      parser: json

      processors:
        logs:
          - name: content_modifier
            action: upsert
            key: my_new_key
            value: 123
  filters:
    - name: grep
      match: '*'
      regex: key pattern

  outputs:
    - name: stdout
      match: '*'

Here is a more complete example with multiple processors:

service:
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2021

pipeline:
  inputs:
    - name: random
      tag: test-tag
      interval_sec: 1
      processors:
        logs:
          - name: modify
            add: hostname monox
          - name: lua
            call: append_tag
            code: |
              function append_tag(tag, timestamp, record)
                 new_record = record
                 new_record["tag"] = tag
                 return 1, timestamp, new_record
              end

  outputs:
    - name: stdout
      match: '*'
      processors:
        logs:
          - name: lua
            call: add_field
            code: |
              function add_field(tag, timestamp, record)
                 new_record = record
                 new_record["output"] = "new data"
                 return 1, timestamp, new_record
              end

You might noticed that processors not only can be attached to input, but also to an output.

How Are Processors Different from Filters?

While processors and filters are similar in that they can transform, enrich, or drop data from the pipeline, there is a significant difference in how they operate:

• Processors: Run in the same thread as the input plugin when the input plugin is configured to be threaded (threaded: true). This design provides better performance, especially in multi-threaded setups.

• Filters: Run in the main event loop. When multiple filters are used, they can introduce performance overhead, particularly under heavy workloads.

Running Filters as Processors

You can configure existing Filters to run as processors. There are no specific changes needed; you simply use the filter name as if it were a native processor.

Example of a Filter Running as a Processor

In the example below, the grep filter is used as a processor to filter log events based on a pattern:

parsers:
  - name: json
    format: json

pipeline:
  inputs:
    - name: tail
      path: /var/log/example.log
      parser: json

      processors:
        logs:
          - name: grep
            regex: log aa
  outputs:
    - name: stdout
      match: '*'

While Fluent Bit comes with a variety of built-in plugins, it also supports loading external plugins at runtime. This feature is especially useful for loading Go or Wasm plugins that are built as shared object files (.so). Fluent Bit's YAML configuration provides two ways to load these external plugins:

1. Inline YAML Section

You can specify external plugins directly within your main YAML configuration file using the plugins section. Here’s an example:

plugins:
  - /path/to/out_gstdout.so

service:
  log_level: info

pipeline:
  inputs:
    - name: random

  outputs:
    - name: gstdout
      match: '*'

2. YAML Plugins File Included via plugins_file Option

Alternatively, you can load external plugins from a separate YAML file by specifying the plugins_file option in the service section. Here’s how to configure this:

service:
  log_level: info
  plugins_file: extra_plugins.yaml

pipeline:
  inputs:
    - name: random

  outputs:
    - name: gstdout
      match: '*'

In this setup, the extra_plugins.yaml file might contain the following plugins section:

plugins:
  - /other/path/to/out_gstdout.so

Key Points

• Built-in vs. External: Fluent Bit comes with many built-in plugins, but you can load external plugins at runtime to extend the tool’s functionality.

• Loading Mechanism: External plugins must be shared object files (.so). You can define them inline in the main YAML configuration or include them from a separate YAML file for better modularity.

The Upstream Servers section defines a group of endpoints, referred to as nodes, which are used by output plugins to distribute data in a round-robin fashion. This is particularly useful for plugins that require load balancing when sending data. Examples of plugins that support this capability include Forward and Elasticsearch.

In YAML, this section is named upstream_servers and requires specifying a name for the group and a list of nodes. Below is an example that defines two upstream server groups: forward-balancing and forward-balancing-2:

upstream_servers:
  - name: forward-balancing
    nodes:
      - name: node-1
        host: 127.0.0.1
        port: 43000

      - name: node-2
        host: 127.0.0.1
        port: 44000

      - name: node-3
        host: 127.0.0.1
        port: 45000
        tls: true
        tls_verify: false
        shared_key: secret

  - name: forward-balancing-2
    nodes:
      - name: node-A
        host: 192.168.1.10
        port: 50000

      - name: node-B
        host: 192.168.1.11
        port: 51000

Key Concepts

• Nodes: Each node in the upstream_servers group must specify a name, host, and port. Additional settings like tls, tls_verify, and shared_key can be configured as needed for secure communication.

Usage Note

While the upstream_servers section can be defined globally, some output plugins may require the configuration to be specified in a separate YAML file. Be sure to consult the documentation for each specific output plugin to understand its requirements.

For more details, refer to the documentation of the respective output plugins.

The env section allows you to define environment variables directly within the configuration file. These variables can then be used to dynamically replace values throughout your configuration using the ${VARIABLE_NAME} syntax.

Values set in the env section are case-sensitive. However, as a best practice, we recommend using uppercase names for environment variables. The example below defines two variables, FLUSH_INTERVAL and STDOUT_FMT, which can be accessed in the configuration using ${FLUSH_INTERVAL} and ${STDOUT_FMT}:

env:
  FLUSH_INTERVAL: 1
  STDOUT_FMT: 'json_lines'

service:
  flush: ${FLUSH_INTERVAL}
  log_level: info

pipeline:
  inputs:
    - name: random

  outputs:
    - name: stdout
      match: '*'
      format: ${STDOUT_FMT}

Predefined Variables

Fluent Bit provides a set of predefined environment variables that can be used in your configuration:

External Variables

In addition to variables defined in the configuration file or the predefined ones, Fluent Bit can access system environment variables set in the user space. These external variables can be referenced in the configuration using the same ${VARIABLE_NAME} pattern.

For example, to set the FLUSH_INTERVAL system environment variable to 2 and use it in your configuration:

export FLUSH_INTERVAL=2

In the configuration file, you can then access this value as follows:

service:
  flush: ${FLUSH_INTERVAL}
  log_level: info

pipeline:
  inputs:
    - name: random

  outputs:
    - name: stdout
      match: '*'
      format: json_lines

This approach allows you to easily manage and override configuration values using environment variables, providing flexibility in various deployment environments.

The includes section allows you to specify additional YAML configuration files to be merged into the current configuration. These files are identified as a list of filenames and can include relative or absolute paths. If no absolute path is provided, the file is assumed to be located in a directory relative to the file that references it.

This feature is useful for organizing complex configurations into smaller, manageable files and including them as needed.

Usage

Below is an example demonstrating how to include additional YAML files using relative path references. This is the file system path structure

├── fluent-bit.yaml
├── inclusion-1.yaml
└── subdir
    └── inclusion-2.yaml

The content of fluent-bit.yaml

includes:
  - inclusion-1.yaml
  - subdir/inclusion-2.yaml

Key Points

• Relative Paths: If a path is not specified as absolute, it will be treated as relative to the file that includes it.

• Organized Configurations: Using the includes section helps keep your configuration modular and easier to maintain.

note: Ensure that the included files are formatted correctly and contain valid YAML configurations for seamless integration.