# Overview

Stream Processing is the ability to query continuous data streams while they are still in motion. [Fluent Bit](https://fluentbit.io) implements a Streaming SQL Engine that can be used for such process.

In order to understand how Stream Processing works in Fluent Bit, we will go through a quick overview of Fluent Bit architecture and how the data goes through the pipeline.

## Fluent Bit Data Pipeline

[Fluent Bit](https://fluentbit.io) collects and process logs (records) from different input sources and allows to parse and filter these records before they hit the Storage interface. Once data is processed and it's in a safe state (either in memory or the file system), the records are routed through the proper output destinations.

> Most of the phases in the pipeline are implemented through plugins: Input, Filter and Output.

![](https://4098349173-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FTNMkuXCv4dctTTuqnfYK%2Fuploads%2Fgit-blob-042d98aaecdaea26920244114f842e11ea64261f%2Fflb_pipeline.png?alt=media)

The Filtering interface is good to perform specific record modifications like append or remove a key, enrich with specific metadata (e.g: Kubernetes Filter) or discard records based on specific conditions. Just after the data will not have any further modification and hits the Storage, optionally, will be redirected to the Stream Processor.

## Stream Processor

The Stream Processor is an independent subsystem that check for new records hitting the Storage interface. By configuration the Stream Processor will attach to records coming from a specific Input plugin (stream) or by applying Tag and Matching rules.

> Every *Input* instance is considered a **Stream**, that stream collects data and ingest records into the pipeline.

![](https://4098349173-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FTNMkuXCv4dctTTuqnfYK%2Fuploads%2Fgit-blob-67e4a80805c98f7ee77c3c40ac5e32365022ebf1%2Fflb_pipeline_sp.png?alt=media)

By configuring specific SQL queries (Structured Query Language), the user can perform specific tasks like key selections, filtering and data aggregation within others. Note that there is **no** database concept here, everything is **schema-less** and happens **in-memory**, for hence the concept of *Tables* as in common relational databases don't exists.

One of the powerful features of Fluent Bit Stream Processor is that allows to create new streams of data using the results from a previous SQL query, these results are re-ingested back into the pipeline to be consumed again for the Stream Processor (if desired) or routed to output destinations such any common record by using Tag/Matching rules (tip: stream processor results can be Tagged!)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.fluentbit.io/manual/3.2/stream-processing/overview.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.