1 of 4

Processors

Content Modifier

The content_modifier processor allows you to manipulate the metadata/attributes and content of Logs and Traces.

Similar to the functionality exposed by filters, this processor presents a unified mechanism to perform such operations for data manipulation. The most significant difference is that processors perform better than filters, and when chaining them, there are no encoding/decoding performance penalties.

Note that processors and this specific component can only be enabled using the new YAML configuration format. Classic mode configuration format doesn't support processors.

Configuration Parameters

Key

Description

Actions

The actions specify the type of operation to run on top of a specific key or content from a Log or a Trace. The following actions are available:

Action

Description

Insert example

The following example appends the key color with the value blue to the log stream.

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4"}'

      processors:
        logs:
          - name: content_modifier
            action: insert
            key: "color"
            value: "blue"
  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Upsert example

Update the value of key1 and insert key2:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4"}'

      processors:
        logs:
          - name: content_modifier
            action: upsert
            key: "key1"
            value: "5678"

          - name: content_modifier
            action: upsert
            key: "key2"
            value: "example"
            
  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Delete example

Delete key2 from the stream:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "key2": "example"}'

      processors:
        logs:
          - name: content_modifier
            action: delete
            key: "key2"   
         
  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Rename example

Change the name of key2 to test:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "key2": "example"}'

      processors:
        logs:
          - name: content_modifier
            action: rename
            key: "key2"
            value: "test"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Hash example

Apply the SHA-256 algorithm for the value of the key password:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"username": "bob", "password": "12345"}'

      processors:
        logs:
          - name: content_modifier
            action: hash
            key: "password"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Extract example

By using a domain address, perform a extraction of the components of it as a list of key value pairs:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"http.url": "https://fluentbit.io/docs?q=example"}'

      processors:
        logs:
          - name: content_modifier
            action: extract
            key: "http.url"
            pattern: ^(?<http_protocol>https?):\/\/(?<http_domain>[^\/\?]+)(?<http_path>\/[^?]*)?(?:\?(?<http_query_params>.*))?
            
  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Convert example

Both keys in the example are strings. Convert the key1 to a double/float type and key2 to a boolean:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "key2": "true"}'

      processors:
        logs:
          - name: content_modifier
            action: convert
            key: key1
            converted_type: int

          - name: content_modifier
            action: convert
            key: key2
            converted_type: boolean
            
  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Metrics Selector

The metric_selector processor allows you to select metrics to include or exclude (similar to the grep filter for logs).

Configuration Parameters

The native processor plugin supports the following configuration parameters:

Key

Description

Default

Configuration Examples

Here is a basic configuration example.

service:
  flush: 5
  daemon: off
  log_level: info

pipeline:
  inputs:
    - name: fluentbit_metrics
      tag: fluentbit.metrics
      scrape_interval: 10

      processors:
        metrics:
          - name: metrics_selector
            metric_name: /storage/
            action: include
          - name: metrics_selector
            metric_name: /fs/
            action: exclude

          - name: labels
            delete: name


  outputs:
    - name: stdout
      match: '*'

All processors are only valid with the YAML configuration format. Processor configuration should be located under the relevant input or output plugin configuration.

Metric_Name parameter will translate the strings which is quoted with backslashes /.../ as Regular expressions. Without them, users need to specify Operation_Type whether prefix matching or substring matching. The default operation is prefix matching. For example, /chunks/ will be translated as a regular expression.

SQL

The sql processor provides a simple interface to select content from Logs by also supporting conditional expressions.

Our SQL processor does not depend on a database or indexing; it runs everything on the fly (this is good). We don't have the concept of tables but you run the query on the STREAM.

Note that this processor differs from the "stream processor interface" that runs after the filters; this one can only be used in the processor's section of the input plugins when using YAML configuration mode.

Configuration Parameters

Key

Description

Simple selection example

The following example generates a sample message with two keys called key and http.url. By using a simple SQL statement we will select only the key http.url.

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "http.url": "https://fluentbit.io/search?q=docs"}'

      processors:
        logs:
          - name: sql
            query: "SELECT http.url FROM STREAM;"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Extract and select example

Similar to the example above, now we will extract the parts of http.url and only select the domain from the value, for that we will use together content-modifier and sql processors together:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "http.url": "https://fluentbit.io/search?q=docs"}'

      processors:
        logs:
          - name: content_modifier
            action: extract
            key: "http.url"
            pattern: ^(?<http_protocol>https?):\/\/(?<http_domain>[^\/\?]+)(?<http_path>\/[^?]*)?(?:\?(?<http_query_params>.*))?

          - name: sql
            query: "SELECT http_domain FROM STREAM;"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

the expected output of this pipeline will be something like this:

{
  "date": 1711059261.630668,
  "http_domain": "fluentbit.io"
}

SQL

The sql processor provides a simple interface to select content from Logs by also supporting conditional expressions.

Our SQL processor does not depend on a database or indexing; it runs everything on the fly (this is good). We don't have the concept of tables but you run the query on the STREAM.

Configuration Parameters

Key

Description

Simple selection example

The following example generates a sample message with two keys called key and http.url. By using a simple SQL statement we will select only the key http.url.

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "http.url": "https://fluentbit.io/search?q=docs"}'

      processors:
        logs:
          - name: sql
            query: "SELECT http.url FROM STREAM;"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Extract and select example

Similar to the example above, now we will extract the parts of http.url and only select the domain from the value, for that we will use together content-modifier and sql processors together:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "http.url": "https://fluentbit.io/search?q=docs"}'

      processors:
        logs:
          - name: content_modifier
            action: extract
            key: "http.url"
            pattern: ^(?<http_protocol>https?):\/\/(?<http_domain>[^\/\?]+)(?<http_path>\/[^?]*)?(?:\?(?<http_query_params>.*))?

          - name: sql
            query: "SELECT http_domain FROM STREAM;"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

the expected output of this pipeline will be something like this:

{
  "date": 1711059261.630668,
  "http_domain": "fluentbit.io"
}

Content Modifier

The content_modifier processor allows you to manipulate the metadata/attributes and content of Logs and Traces.

Note that processors and this specific component can only be enabled using the new YAML configuration format. Classic mode configuration format doesn't support processors.

Configuration Parameters

Key

Description

Actions

The actions specify the type of operation to run on top of a specific key or content from a Log or a Trace. The following actions are available:

Action

Description

Insert example

The following example appends the key color with the value blue to the log stream.

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4"}'

      processors:
        logs:
          - name: content_modifier
            action: insert
            key: "color"
            value: "blue"
  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Upsert example

Update the value of key1 and insert key2:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4"}'

      processors:
        logs:
          - name: content_modifier
            action: upsert
            key: "key1"
            value: "5678"

          - name: content_modifier
            action: upsert
            key: "key2"
            value: "example"
            
  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Delete example

Delete key2 from the stream:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "key2": "example"}'

      processors:
        logs:
          - name: content_modifier
            action: delete
            key: "key2"   
         
  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Rename example

Change the name of key2 to test:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "key2": "example"}'

      processors:
        logs:
          - name: content_modifier
            action: rename
            key: "key2"
            value: "test"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Hash example

Apply the SHA-256 algorithm for the value of the key password:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"username": "bob", "password": "12345"}'

      processors:
        logs:
          - name: content_modifier
            action: hash
            key: "password"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Extract example

By using a domain address, perform a extraction of the components of it as a list of key value pairs:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"http.url": "https://fluentbit.io/docs?q=example"}'

      processors:
        logs:
          - name: content_modifier
            action: extract
            key: "http.url"
            pattern: ^(?<http_protocol>https?):\/\/(?<http_domain>[^\/\?]+)(?<http_path>\/[^?]*)?(?:\?(?<http_query_params>.*))?
            
  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Convert example

Both keys in the example are strings. Convert the key1 to a double/float type and key2 to a boolean:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "key2": "true"}'

      processors:
        logs:
          - name: content_modifier
            action: convert
            key: key1
            converted_type: int

          - name: content_modifier
            action: convert
            key: key2
            converted_type: boolean
            
  outputs:
    - name : stdout
      match: '*'
      format: json_lines