Fluent Bit: Official Manual
SlackGitHubCommunity MeetingsSandbox and LabsWebinars
4.0
4.0
  • Fluent Bit Documentation
  • About
    • What is Fluent Bit?
    • A Brief History of Fluent Bit
    • Fluentd and Fluent Bit
    • License
    • Sandbox and Lab Resources
  • Concepts
    • Key Concepts
    • Buffering
    • Data Pipeline
      • Input
      • Parser
      • Filter
      • Buffer
      • Router
      • Output
  • Installation
    • Getting Started with Fluent Bit
    • Upgrade Notes
    • Supported Platforms
    • Requirements
    • Sources
      • Download Source Code
      • Build and Install
      • Build with Static Configuration
    • Linux Packages
      • Amazon Linux
      • Redhat / CentOS
      • Debian
      • Ubuntu
      • Raspbian / Raspberry Pi
    • Docker
    • Containers on AWS
    • Amazon EC2
    • Kubernetes
    • macOS
    • Windows
    • Yocto / Embedded Linux
    • Buildroot / Embedded Linux
  • Administration
    • Configuring Fluent Bit
      • YAML Configuration
        • Service
        • Parsers
        • Multiline Parsers
        • Pipeline
        • Plugins
        • Upstream Servers
        • Environment Variables
        • Includes
      • Classic mode
        • Format and Schema
        • Configuration File
        • Variables
        • Commands
        • Upstream Servers
        • Record Accessor
      • Unit Sizes
      • Multiline Parsing
    • Transport Security
    • Buffering and Storage
    • Backpressure
    • Scheduling and Retries
    • Networking
    • Memory Management
    • Monitoring
    • Multithreading
    • HTTP Proxy
    • Hot Reload
    • Troubleshooting
    • Performance Tips
    • AWS credentials
  • Local Testing
    • Validating your Data and Structure
    • Running a Logging Pipeline Locally
  • Data Pipeline
    • Pipeline Monitoring
    • Inputs
      • Collectd
      • CPU Log Based Metrics
      • Disk I/O Log Based Metrics
      • Docker Events
      • Docker Log Based Metrics
      • Dummy
      • Elasticsearch
      • Exec
      • Exec Wasi
      • Ebpf
      • Fluent Bit Metrics
      • Forward
      • Head
      • Health
      • HTTP
      • Kafka
      • Kernel Logs
      • Kubernetes Events
      • Memory Metrics
      • MQTT
      • Network I/O Log Based Metrics
      • NGINX Exporter Metrics
      • Node Exporter Metrics
      • OpenTelemetry
      • Podman Metrics
      • Process Exporter Metrics
      • Process Log Based Metrics
      • Prometheus Remote Write
      • Prometheus Scrape Metrics
      • Random
      • Serial Interface
      • Splunk
      • Standard Input
      • StatsD
      • Syslog
      • Systemd
      • Tail
      • TCP
      • Thermal
      • UDP
      • Windows Event Log
      • Windows Event Log (winevtlog)
      • Windows Exporter Metrics
    • Parsers
      • Configuring Parser
      • JSON
      • Regular Expression
      • LTSV
      • Logfmt
      • Decoders
    • Processors
      • Content Modifier
      • Labels
      • Metrics Selector
      • OpenTelemetry Envelope
      • Sampling
      • SQL
      • Filters as processors
      • Conditional processing
    • Filters
      • AWS Metadata
      • CheckList
      • ECS Metadata
      • Expect
      • GeoIP2 Filter
      • Grep
      • Kubernetes
      • Log to Metrics
      • Lua
      • Parser
      • Record Modifier
      • Modify
      • Multiline
      • Nest
      • Nightfall
      • Rewrite Tag
      • Standard Output
      • Sysinfo
      • Throttle
      • Type Converter
      • Tensorflow
      • Wasm
    • Outputs
      • Amazon CloudWatch
      • Amazon Kinesis Data Firehose
      • Amazon Kinesis Data Streams
      • Amazon S3
      • Azure Blob
      • Azure Data Explorer
      • Azure Log Analytics
      • Azure Logs Ingestion API
      • Counter
      • Dash0
      • Datadog
      • Dynatrace
      • Elasticsearch
      • File
      • FlowCounter
      • Forward
      • GELF
      • Google Chronicle
      • Google Cloud BigQuery
      • HTTP
      • InfluxDB
      • Kafka
      • Kafka REST Proxy
      • LogDNA
      • Loki
      • Microsoft Fabric
      • NATS
      • New Relic
      • NULL
      • Observe
      • OpenObserve
      • OpenSearch
      • OpenTelemetry
      • Oracle Log Analytics
      • PostgreSQL
      • Prometheus Exporter
      • Prometheus Remote Write
      • SkyWalking
      • Slack
      • Splunk
      • Stackdriver
      • Standard Output
      • Syslog
      • TCP and TLS
      • Treasure Data
      • Vivo Exporter
      • WebSocket
  • Stream Processing
    • Introduction to Stream Processing
    • Overview
    • Changelog
    • Getting Started
      • Fluent Bit + SQL
      • Check Keys and NULL values
      • Hands On 101
  • Fluent Bit for Developers
    • C Library API
    • Ingest Records Manually
    • Golang Output Plugins
    • WASM Filter Plugins
    • WASM Input Plugins
    • Developer guide for beginners on contributing to Fluent Bit
Powered by GitBook
On this page
  • Statements
  • SELECT
  • CREATE STREAM
  • Aggregation functions
  • AVG
  • COUNT
  • MIN
  • MAX
  • SUM
  • Time Functions
  • NOW
  • UNIX_TIMESTAMP
  • Record Functions
  • RECORD_TAG
  • RECORD_TIME
  • WHERE condition

Was this helpful?

Export as PDF
  1. Stream Processing
  2. Getting Started

Fluent Bit + SQL

Last updated 16 days ago

Was this helpful?

Stream processing in Fluent Bit uses SQL to perform record queries.

For more information, see the .

Statements

Use the following SQL statements in Fluent Bit.

SELECT

SELECT results_statement
  FROM STREAM:stream_name | TAG:match_rule
  [WINDOW TUMBLING (integer SECOND)]
  [WHERE condition]
  [GROUP BY groupby]

Groups keys from records that originate from a specified stream, or from records that match a specific tag pattern.

A SELECT statement not associated with stream creation will send the results to the standard output interface, which can be helpful for debugging purposes.

You can filter the results of this query by applying a condition by using a WHERE statement. For information about the WINDOW and GROUP BY statements, see .

Examples

Selects all keys from records that originate from a stream called apache:

SELECT * FROM STREAM:apache;

Selects the code key from records with tags whose name begins with apache:

SELECT code AS http_status FROM TAG:'apache.*';

CREATE STREAM

CREATE STREAM stream_name
  [WITH (property_name=value, [...])]
  AS select_statement

Creates a new stream of data using the results from a SELECT statement. If the Tag property in the WITH statement is set, this new stream can optionally be re-ingested into the Fluent Bit pipeline.

Examples

Creates a new stream called hello_ from a stream called apache:

CREATE STREAM hello AS SELECT * FROM STREAM:apache;

Creates a new stream called hello for all records whose original tag name begins with apache:

CREATE STREAM hello AS SELECT * FROM TAG:'apache.*';

Aggregation functions

You can use aggregation functions in the results_statement on keys, which lets you perform data calculation on groups of records. These groups are determined by the WINDOW key. If WINDOW is unspecified, aggregation functions are applied to the current buffer of records received, which might have a non-deterministic number of elements. You can also apply aggregation functions to records in a window of a specific time interval.

Fluent Bit uses a tumbling window, which is non-overlapping. For example, a window size of 5 performs aggregation computations on records during a five-second interval, then starts new calculations for the next interval.

Additionally, you can use the GROUP BY statement to group results by one or more keys with matching values.

AVG

SELECT AVG(size) FROM STREAM:apache WHERE method = 'POST' ;

Calculates the average size of POST requests.

COUNT

SELECT host, COUNT(*) FROM STREAM:apache WINDOW TUMBLING (X SECOND) GROUP BY host;

Counts the number of records in a five-second window, grouped by host IP addresses.

MIN

SELECT MIN(key) FROM STREAM:apache;

Returns the minimum value of a key in a set of records.

MAX

SELECT MAX(key) FROM STREAM:apache;

Returns the maximum value of a key in a set of records.

SUM

SELECT SUM(key) FROM STREAM:apache;

Calculates the sum of all values of a key in a set of records.

Time Functions

Use time functions to add a new key with time data into a record.

NOW

SELECT NOW() FROM STREAM:apache;

Adds the current system time to a record using the format %Y-%m-%d %H:%M:%S. Output example: 2019-03-09 21:36:05.

UNIX_TIMESTAMP

SELECT UNIX_TIMESTAMP() FROM STREAM:apache;

Adds the current Unix time to a record. Output example: 1552196165.

Record Functions

Use record functions to append new keys to a record using values from the record's context.

RECORD_TAG

SELECT RECORD_TAG() FROM STREAM:apache;

Append tag string associated to the record as a new key.

RECORD_TIME

SELECT RECORD_TIME() FROM STREAM:apache;

WHERE condition

Similar to conventional SQL statements, Fluent Bit supports the WHERE condition. You can use this condition in both keys and subkeys. For example:

SELECT AVG(size) FROM STREAM:apache WHERE method = 'POST' AND status = 200;

You can confirm whether a key exists in a record by using the record-specific function @record.contains:

SELECT MAX(key) FROM STREAM:apache WHERE @record.contains(key);

To determine if the value of a key is NULL:

SELECT MAX(key) FROM STREAM:apache WHERE key IS NULL;

Or similar:

SELECT * FROM STREAM:apache WHERE user IS NOT NULL;
stream processing README file
Aggregation functions