Stream Processor
Fluent Bit v1.1 comes with a new and optional Stream Processor Engine that allows to do data processing through SQL queries. This article covers the format of the expected configuration file.
For more details about the Stream Processor Engine use please refer to the following guide:
Concepts
The stream processor can be configured through defining Tasks which have a name and an execution SQL statement:
Streams File Configuration
The Stream Processor is configured through a streams file that is referenced from the main fluent-bit.conf configuration file through the Streams_File key. The content of the streams file must have the following format specified in the table below:
Configuration Example
Consider the following fluent-bit.conf configuration file:
Now creates a stream_processor.conf configuration file with the following content:
On the query there are a few things happening:
Fluent Bit will gather CPU usage metrics through CPU input plugin (metrics are calculated by default every second).
Stream Processor have a Task attached to any incoming Stream of data called cpu_data (check the alias set in the Input section).
Stream Processor will aggregate the value of cpu_p record field and calculate it average during a window of 5 seconds.
Stream Processor every 5 seconds will send the results back into Fluent Bit pipeline with a tag called results.
Fluent Bit output section will match results tagged records and print them to the standard output interface.
You should see the following output in your terminal:
If you want to learn more about our Stream Processor engine please read the official guide.
Last updated