Processors

Processors are components that modify, transform, or enhance the data that flows through Fluent Bit.

Each input plugin or output plugin can have one or more attached processors. Processors can modify logs, metrics, and traces.

Only YAML configuration files support processors.

Available processors

Fluent Bit offers the following processors:

• Content modifier: Manipulate the content, metadata, and attributes of logs and traces.

• Labels: Add, update, or delete metric labels.

• Metrics selector: Choose which metrics to keep or discard.

• OpenTelemetry envelope: Transform logs into an OpenTelemetry-compatible format.

• Sampling: Apply head or tail sampling to incoming traces.

• SQL: Use SQL queries to extract log content.

• Filters as processors: Use filters as processors.

Features

Compatible processors include the following features:

• Conditional processing: Selectively apply processors to logs based on the value of fields that those logs contain.

How processors are different from filters

Although processors and filters both transform data, they're different in many ways.

Processors are attached to individual input and output plugins, and don't use tag matching. Filters are defined globally, and do use tag matching.

Processors run in the same thread as their associated input or output plugin. This lets processors execute immediately and helps reduce performance bottlenecks, especially when multithreading is enabled. Filters always run in the main thread, and using multiple filters can introduce performance overhead, particularly under heavy workloads.

Additionally, filters can be implemented in a way that mimics the behavior of processors, but processors can't be implemented in a way that mimics filters.

Example configuration

In the following example, the content modifier processor inserts or updates (upserts) the key my_new_key with the value 123 for all log records generated by the tail plugin. This processor is only applied to logs.

fluent-bit.yaml

parsers:
    - name: json
      format: json

pipeline:
      inputs:
          - name: tail
            path: /var/log/example.log
            parser: json

            processors:
                logs:
                    - name: content_modifier
                      action: upsert
                      key: my_new_key
                      value: 123

      filters:
          - name: grep
            match: '*'
            regex: key pattern

      outputs:
          - name: stdout
            match: '*'