# Fluent Bit + SQL Fluent Bit stream processor uses common SQL to perform record queries. The following section describe the features available and examples of it. ## Statements You can find the detailed query language syntax in BNF form [here](https://github.com/fluent/fluent-bit/tree/master/src/stream_processor). The following section will be a brief introduction on how to write SQL queries for Fluent Bit stream processing. ### SELECT Statement #### Synopsis ```sql SELECT results_statement FROM STREAM:stream_name | TAG:match_rule [WINDOW TUMBLING (integer SECOND)] [WHERE condition] [GROUP BY groupby] ``` #### Description Select keys from records coming from a stream or records matching a specific Tag pattern. Note that a simple `SELECT` statement **not** associated from a stream creation will send the results to the standard output interface (stdout), useful for debugging purposes. The query allows filtering the results by applying a condition using `WHERE` statement. We will explain `WINDOW` and `GROUP BY` statements later in aggregation functions section. #### Examples Select all keys from records coming from a stream called *apache*: ```sql SELECT * FROM STREAM:apache; ``` Select code key from records which Tag starts with *apache.*: ```sql SELECT code AS http_status FROM TAG:'apache.*'; ``` > Since the TAG selector allows the use of wildcards, we put the value between single quotes. ### CREATE STREAM Statement #### Synopsis ```sql CREATE STREAM stream_name [WITH (property_name=value, [...])] AS select_statement ``` #### Description Create a new stream of data using the results from the `SELECT` statement. New stream created can be optionally re-ingested back into Fluent Bit pipeline if the property *Tag* is set in the WITH statement. #### Examples Create a new stream called *hello* from stream called *apache*: ```sql CREATE STREAM hello AS SELECT * FROM STREAM:apache; ``` Create a new stream called hello for all records which original Tag starts with *apache*: ```sql CREATE STREAM hello AS SELECT * FROM TAG:'apache.*'; ``` ## Aggregation Functions Aggregation functions are used in `results_statement` on the keys, allowing to perform data calculation on groups of records. Group of records that aggregation functions apply on are determined by `WINDOW` keyword. When `WINDOW` is not specified, aggregation functions apply on the current buffer of records received, which may have non-deterministic number of elements. Aggregation functions can be applied on records in a window of a specific time interval (see the syntax of `WINDOW` in select statement). Fluent Bit streaming currently supports tumbling window, which is non-overlapping window type. That means, a window of size 5 seconds performs aggregation computations on records over a 5-second interval, and then starts new calculations for the next interval. In addition, the syntax support `GROUP BY` statement, which groups the results by the one or more keys, when they have the same values. ### AVG #### Synopsis ```sql SELECT AVG(size) FROM STREAM:apache WHERE method = 'POST' ; ``` #### Description Calculates the average of request sizes in POST requests. ### COUNT #### Synopsis ```sql SELECT host, COUNT(*) FROM STREAM:apache WINDOW TUMBLING (5 SECOND) GROUP BY host; ``` #### Description Count the number of records in 5 second windows group by host IP addresses. ### MIN #### Synopsis ```sql SELECT MIN(key) FROM STREAM:apache; ``` #### Description Gets the minimum value of a key in a set of records. ### MAX #### Synopsis ```sql SELECT MIN(key) FROM STREAM:apache; ``` #### Description Gets the maximum value of a key in a set of records. ### SUM #### Synopsis ```sql SELECT SUM(key) FROM STREAM:apache; ``` #### Description Calculates the sum of all values of key in a set of records. ## Time Functions Time functions adds a new key into the record with timing data ### NOW #### Synopsis ```sql SELECT NOW() FROM STREAM:apache; ``` #### Description Add system time using format: %Y-%m-%d %H:%M:%S. Output example: 2019-03-09 21:36:05. ### UNIX\_TIMESTAMP #### Synopsis ```sql SELECT UNIX_TIMESTAMP() FROM STREAM:apache; ``` #### Description Add current Unix timestamp to the record. Output example: 1552196165 . ## Record Functions Record functions append new keys to the record using values from the record context. ### RECORD\_TAG #### Synopsis ```sql SELECT RECORD_TAG() FROM STREAM:apache; ``` #### Description Append Tag string associated to the record as a new key. ### RECORD\_TIME #### Synopsis ```sql SELECT RECORD_TIME() FROM STREAM:apache; ``` ## WHERE Condition Similar to conventional SQL statements, `WHERE` condition is supported in Fluent Bit query language. The language supports conditions over keys and subkeys, for instance: ```sql SELECT AVG(size) FROM STREAM:apache WHERE method = 'POST' AND status = 200; ``` It is possible to check the existence of a key in the record using record-specific function `@record.contains`: ```sql SELECT MAX(key) FROM STREAM:apache WHERE @record.contains(key); ``` And to check if the value of a key is/is not `NULL`: ```sql SELECT MAX(key) FROM STREAM:apache WHERE key IS NULL; ``` ```sql SELECT * FROM STREAM:apache WHERE user IS NOT NULL; ``` #### Description Append a new key with the record Timestamp in *double* format: seconds.nanoseconds. Output example: 1552196165.705683 . --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.fluentbit.io/manual/3.2/stream-processing/getting-started/fluent-bit-sql.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.