SQL

The SQL processor lets you use conditional expressions to select content from logs. This processor doesn't depend on a database or table. Instead, your queries run on the stream.

This processor differs from the stream processor interface that runs after filters.

Only YAML configuration files support processors.

Configuration parameters

Key	Description
query	The SQL statement to query your logs stream. This statement must end with ;.

Basic selection example

The following example generates a sample message with the keys key and http.url, and then uses a SQL statement to select only the key http.url.

fluent-bit.yaml

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "http.url": "https://fluentbit.io/search?q=docs"}'

      processors:
        logs:
          - name: sql
            query: "SELECT http.url FROM STREAM;"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

Extract and select example

The following example is similar to the previous example, but additionally extracts part of http.url to select the domain from the value. To accomplish this, use the content-modifier and sql processors in tandem:

fluent-bit.yaml

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "http.url": "https://fluentbit.io/search?q=docs"}'

      processors:
        logs:
          - name: content_modifier
            action: extract
            key: "http.url"
            pattern: ^(?<http_protocol>https?):\/\/(?<http_domain>[^\/\?]+)(?<http_path>\/[^?]*)?(?:\?(?<http_query_params>.*))?

    - name: sql
      query: "SELECT http_domain FROM STREAM;"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

The resulting output resembles the following:

{
  "date": 1711059261.630668,
  "http_domain": "fluentbit.io"
}