The sql processor provides a simple interface to select content from Logs by also supporting conditional expressions.

Our SQL processor does not depend on a database or indexing; it runs everything on the fly (this is good). We don't have the concept of tables but you run the query on the STREAM.

Note that this processor differs from the "stream processor interface" that runs after the filters; this one can only be used in the processor's section of the input plugins when using YAML configuration mode.

Configuration Parameters

Simple selection example

The following example generates a sample message with two keys called key and http.url. By using a simple SQL statement we will select only the key http.url.

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "http.url": "https://fluentbit.io/search?q=docs"}'

      processors:
        logs:
          - name: sql
            query: "SELECT http.url FROM STREAM;"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Extract and select example

Similar to the example above, now we will extract the parts of http.url and only select the domain from the value, for that we will use together content-modifier and sql processors together:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "http.url": "https://fluentbit.io/search?q=docs"}'

      processors:
        logs:
          - name: content_modifier
            action: extract
            key: "http.url"
            pattern: ^(?<http_protocol>https?):\/\/(?<http_domain>[^\/\?]+)(?<http_path>\/[^?]*)?(?:\?(?<http_query_params>.*))?

          - name: sql
            query: "SELECT http_domain FROM STREAM;"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

the expected output of this pipeline will be something like this:

{
  "date": 1711059261.630668,
  "http_domain": "fluentbit.io"
}

The OpenTelemetry Envelope processor is used to transform your data to be compatible with the OpenTelemetry Log schema. If your data was not generated by OpenTelemetry input and your backend or destination for your logs expects to be in an OpenTelemetry schema.

Configuration Parameters

The processor does not provide any extra configuration parameter, it can be used directly in your processors Yaml directive.

Usage Example

In this example, we will use the Dummy input plugin to generate a sample message per second, right after is created the processor opentelemetry_envelope is used to transform the data to be compatible with the OpenTelemetry Log schema. The output is sent to the standard output and also to an OpenTelemetry collector which is receiving data in port 4318.

fluent-bit.yaml

service:
  flush: 1
  log_level: info

pipeline:
  inputs:
    - name: dummy
      dummy: '{"message": "Hello World"}'

      processors:
        logs:
          - name: opentelemetry_envelope

  outputs:
    - name : stdout
      match: '*'

    - name: opentelemetry
      match: '*'
      host: 127.0.0.1
      port: 4318

otel-collector.yaml

receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4318

exporters:
  file:
    path: out.json
  logging:
    loglevel: info

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [file, logging]

You will notice in the standard output of FLuent Bit will print the raw representation of the schema, however, the OpenTelemetry collector will receive the data in the OpenTelemetry Log schema.

Inspecting the output file out.json you will see the data in the OpenTelemetry Log schema:

{
  "resourceLogs": [
    {
      "resource": {},
      "scopeLogs": [
        {
          "scope": {},
          "logRecords": [
            {
              "timeUnixNano": "1722904188085758000",
              "body": {
                "stringValue": "dummy"
              },
              "traceId": "",
              "spanId": ""
            }
          ]
        }
      ]
    }
  ]
}

While OpenTelemetry Envelope enrich your logs with the Schema, you might be interested into take a step further and use the Content Modifier processor to modify the content of your logs. Here is a quick example that will allow you to add some resource and scope attributes to your logs:

service:
  flush: 1
  log_level: info

pipeline:
  inputs:
    - name: dummy
      dummy: '{"message": "Hello World"}'

      processors:
        logs:
          - name: opentelemetry_envelope

          - name: content_modifier
            context: otel_resource_attributes
            action: upsert
            key: service.name
            value: my-service

  outputs:
    - name : stdout
      match: '*'

    - name: opentelemetry
      match: '*'
      host: 127.0.0.1
      port: 4318

The collector JSON output will look like this:

{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "my-service"
            }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {},
          "logRecords": [
            {
              "timeUnixNano": "1722904465173450000",
              "body": {
                "stringValue": "Hello World"
              },
              "traceId": "",
              "spanId": ""
            }
          ]
        }
      ]
    }
  ]
}

For more details about further processing, read the Content Modifier processor documentation.

The labels processor lets you manipulate the labels of metrics.

Similar to filters, this processor presents a enriching/modifying mechanism to perform operations for labels manipulation. The most significant difference is that processors perform better than filters, and when chaining them there are no encoding or decoding performance penalties.

Configuration Parameters

Update example

Change the value of the name to fluentbit:

pipeline:
  inputs:
    - name: fluentbit_metrics
      processors:
        metrics:
          - name: labels
            update: name fluentbit
  outputs:
    - name : stdout
      match: '*'

Insert example

The following example appends the key agent with the value fluentbit as the label of metrics:

pipeline:
  inputs:
    - name: fluentbit_metrics
      processors:
        metrics:
          - name: labels
            insert: agent fluentbit
  outputs:
    - name : stdout
      match: '*'

Upsert example

Upsert the value of name and insert fluentbit:

pipeline:
  inputs:
    - name: fluentbit_metrics
      processors:
        metrics:
          - name: labels
            upsert: name fluentbit
  outputs:
    - name : stdout
      match: '*'

Delete example

Delete containing name key from metrics:

pipeline:
  inputs:
    - name: fluentbit_metrics
      processors:
        metrics:
          - name: labels
            delete: name
  outputs:
    - name : stdout
      match: '*'

Hash example

Apply the SHA-1 algorithm for the value of the key hostname:

pipeline:
  inputs:
    - name: fluentbit_metrics
      processors:
        metrics:
          - name: labels
            hash: hostname
  outputs:
    - name : stdout
      match: '*'

The metric_selector processor allows you to select metrics to include or exclude (similar to the grep filter for logs).

Configuration Parameters

The native processor plugin supports the following configuration parameters:

Configuration Examples

Here is a basic configuration example.

All processors are only valid with the YAML configuration format. Processor configuration should be located under the relevant input or output plugin configuration.

Metric_Name parameter will translate the strings which is quoted with backslashes /.../ as Regular expressions. Without them, users need to specify Operation_Type whether prefix matching or substring matching. The default operation is prefix matching. For example, /chunks/ will be translated as a regular expression.

The content_modifier processor allows you to manipulate the messages, metadata/attributes and content of Logs and Traces.

Similar to the functionality exposed by filters, this processor presents a unified mechanism to perform such operations for data manipulation. The most significant difference is that processors perform better than filters, and when chaining them, there are no encoding/decoding performance penalties.

Note that processors and this specific component can only be enabled using the new YAML configuration format. Classic mode configuration format doesn't support processors.

Contexts

The processor, works on top of what we call a context, meaning the place where the content modification will happen. We provide different contexts to manipulate the desired information, the following contexts are available:

OpenTelemetry Contexts

In addition, we provide special contexts to operate on data that follows an OpenTelemetry Log Schema, all of them operates on shared data across a group of records:

TIP: if your data is not following the OpenTelemetry Log Schema and your backend or destination for your logs expects to be in an OpenTelemetry schema, take a look at the processor called OpenTelemetry Envelope that you can use in conjunbction with this processor to transform your data to be compatible with OpenTelemetry Log schema.

Configuration Parameters

Actions

The actions specify the type of operation to run on top of a specific key or content from a Log or a Trace. The following actions are available:

Insert example

The following example appends the key color with the value blue to the log stream.

Upsert example

Update the value of key1 and insert key2:

Delete example

Delete key2 from the stream:

Rename example

Change the name of key2 to test:

Hash example

Apply the SHA-256 algorithm for the value of the key password:

Extract example

By using a domain address, perform a extraction of the components of it as a list of key value pairs:

Convert example

Both keys in the example are strings. Convert the key1 to a double/float type and key2 to a boolean: