1.9
Installation
Administration
Data Pipeline
Stream Processing
Hands On! 101
This article goes through very specific and simple steps to learn how Stream Processor works. For simplicity it uses a custom Docker image that contains the relevant components for testing.
Requirements
The following tutorial requires the following software components:
In addition download the following data sample file (130KB):
Stream Processing using the command line
For all next steps we will run Fluent Bit from the command line, and for simplicity we will use the official Docker image.
1. Fluent Bit version:
1
$ docker run -ti fluent/fluent-bit:1.4 /fluent-bit/bin/fluent-bit --version
2
Fluent Bit v1.8.2
Copied!
2. Parse sample files
The samples file contains JSON records. On this command, we are appending the Parsers configuration file and instructing tail input plugin to parse the content as json:
1
$ docker run -ti -v `pwd`/sp-samples-1k.log:/sp-samples-1k.log \
2
fluent/fluent-bit:1.8.2 \
3
/fluent-bit/bin/fluent-bit -R /fluent-bit/etc/parsers.conf \
4
-i tail -p path=/sp-samples-1k.log \
5
-p parser=json \
6
-p read_from_head=true \
7
-o stdout -f 1
Copied!
The command above will simply print the parsed content to the standard output interface. The content will print the Tag associated to each record and an array with two fields: record timestamp and record map:
1
Fluent Bit v1.8.2
2
* Copyright (C) 2019-2021 The Fluent Bit Authors
3
* Copyright (C) 2015-2018 Treasure Data
4
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
5
* https://fluentbit.io
6
​
7
[2019/05/08 13:34:16] [ info] [storage] initializing...
8
[2019/05/08 13:34:16] [ info] [storage] in-memory
9
[2019/05/08 13:34:16] [ info] [storage] normal synchronization mode, checksum disabled
10
[2019/05/08 13:34:16] [ info] [engine] started (pid=1)
11
[2019/05/08 13:34:16] [ info] [sp] stream processor started
12
[0] tail.0: [1557322456.315513208, {"date"=>"22/abr/2019:12:43:51 -0600", "ip"=>"73.113.230.135", "word"=>"balsamine", "country"=>"Japan", "flag"=>false, "num"=>96}]
13
[1] tail.0: [1557322456.315525280, {"date"=>"22/abr/2019:12:43:52 -0600", "ip"=>"242.212.128.227", "word"=>"inappendiculate", "country"=>"Chile", "flag"=>false, "num"=>15}]
14
[2] tail.0: [1557322456.315532364, {"date"=>"22/abr/2019:12:43:52 -0600", "ip"=>"85.61.182.212", "word"=>"elicits", "country"=>"Argentina", "flag"=>true, "num"=>73}]
15
[3] tail.0: [1557322456.315538969, {"date"=>"22/abr/2019:12:43:52 -0600", "ip"=>"124.192.66.23", "word"=>"Dwan", "country"=>"Germany", "flag"=>false, "num"=>67}]
16
[4] tail.0: [1557322456.315545150, {"date"=>"22/abr/2019:12:43:52 -0600", "ip"=>"18.135.244.142", "word"=>"chesil", "country"=>"Argentina", "flag"=>true, "num"=>19}]
17
[5] tail.0: [1557322456.315550927, {"date"=>"22/abr/2019:12:43:52 -0600", "ip"=>"132.113.203.169", "word"=>"fendered", "country"=>"United States", "flag"=>true, "num"=>53}]
Copied!
As of now there is no Stream Processing, on step #3 we will start doing some basic queries.
3. Selecting specific record keys
This command introduces a Stream Processor (SP) query through the -T option and changes the output plugin to null, this is done with the purpose of obtaining the SP results in the standard output interface and avoid confusions in the terminal.
1
$ docker run -ti -v `pwd`/sp-samples-1k.log:/sp-samples-1k.log \
2
fluent/fluent-bit:1.2 \
3
/fluent-bit/bin/fluent-bit \
4
-R /fluent-bit/etc/parsers.conf \
5
-i tail \
6
-p path=/sp-samples-1k.log \
7
-p parser=json \
8
-p read_from_head=true \
9
-T "SELECT word, num FROM STREAM:tail.0 WHERE country='Chile';" \
10
-o null -f 1
Copied!
The query above aims to retrieve all records that a key named country value matches the value Chile, and for each match compose and output a record using only the key fields word and num:
1
[0] [1557322913.263534, {"word"=>"Candide", "num"=>94}]
2
[0] [1557322913.263581, {"word"=>"delightfulness", "num"=>99}]
3
[0] [1557322913.263607, {"word"=>"effulges", "num"=>63}]
4
[0] [1557322913.263690, {"word"=>"febres", "num"=>21}]
5
[0] [1557322913.263706, {"word"=>"decasyllables", "num"=>76}]
Copied!
4. Calculate Average Value
The following query is similar to the one in the previous step, but this time we will use the aggregation function called AVG() to get the average value of the records ingested:
1
$ docker run -ti -v `pwd`/sp-samples-1k.log:/sp-samples-1k.log \
2
fluent/fluent-bit:1.8.2 \
3
/fluent-bit/bin/fluent-bit \
4
-R /fluent-bit/etc/parsers.conf \
5
-i tail \
6
-p path=/sp-samples-1k.log \
7
-p parser=json \
8
-T "SELECT AVG(num) FROM STREAM:tail.0 WHERE country='Chile';" \
9
-o null -f 1
Copied!
output:
1
[0] [1557323573.940149, {"AVG(num)"=>61.230770}]
2
[0] [1557323573.941890, {"AVG(num)"=>47.842106}]
3
[0] [1557323573.943544, {"AVG(num)"=>40.647060}]
4
[0] [1557323573.945086, {"AVG(num)"=>56.812500}]
5
[0] [1557323573.945130, {"AVG(num)"=>99.000000}]
Copied!
why did we get multiple records? Answer: When Fluent Bit processes the data, records come in chunks and the Stream Processor runs the process over chunks of data, so the input plugin ingested 5 chunks of records and SP processed the query for each chunk independently. To process multiple chunks at once we have to group results during windows of time.
5. Grouping Results and Window
Grouping results aims to simplify data processing and when used in a defined window of time we can achieve great things. The next query group the results by country and calculate the average of num value, the processing window is 1 second which basically means: process all incoming chunks coming within 1 second window:
1
$ docker run -ti -v `pwd`/sp-samples-1k.log:/sp-samples-1k.log \
2
fluent/fluent-bit:1.8.2 \
3
/fluent-bit/bin/fluent-bit \
4
-R /fluent-bit/etc/parsers.conf \
5
-i tail \
6
-p path=/sp-samples-1k.log \
7
-p parser=json \
8
-T "SELECT country, AVG(num) FROM STREAM:tail.0 \
9
WINDOW TUMBLING (1 SECOND) \
10
WHERE country='Chile' \
11
GROUP BY country;" \
12
-o null -f 1
Copied!
output:
1
[0] [1557324239.003211, {"country"=>"Chile", "AVG(num)"=>53.164558}]
Copied!
6. Ingest Stream Processor results as new Stream of Data
Now we see a more real-world use case. Sending data results to the standard output interface is good for learning purposes, but now we will instruct the Stream Processor to ingest results as part of Fluent Bit data pipeline and attach a Tag to them.
This can be done using the CREATE STREAM statement that will also tag results with sp-results value. Note that output plugin parameter is now stdout matching all records tagged with sp-results:
1
$ docker run -ti -v `pwd`/sp-samples-1k.log:/sp-samples-1k.log \
2
fluent/fluent-bit:1.8.2 \
3
/fluent-bit/bin/fluent-bit \
4
-R /fluent-bit/etc/parsers.conf \
5
-i tail \
6
-p path=/sp-samples-1k.log \
7
-p parser=json \
8
-p read_from_head=true \
9
-T "CREATE STREAM results WITH (tag='sp-results') \
10
AS \
11
SELECT country, AVG(num) FROM STREAM:tail.0 \
12
WINDOW TUMBLING (1 SECOND) \
13
WHERE country='Chile' \
14
GROUP BY country;" \
15
-o stdout -m 'sp-results' -f 1
Copied!
output:
1
[0] sp-results: [1557325032.000160100, {"country"=>"Chile", "AVG(num)"=>53.164558}]
Copied!
F.A.Q
Where STREAM name comes from?
Fluent Bit have the notion of streams, and every input plugin instance gets a default name. You can override that behavior by setting an alias. Check the alias parameter and new stream name in the following example:
1
$ docker run -ti -v `pwd`/sp-samples-1k.log:/sp-samples-1k.log \
2
fluent/fluent-bit:1.8.2 \
3
/fluent-bit/bin/fluent-bit \
4
-R /fluent-bit/etc/parsers.conf \
5
-i tail \
6
-p path=/sp-samples-1k.log \
7
-p parser=json \
8
-p read_from_head=true \
9
-p alias=samples \
10
-T "CREATE STREAM results WITH (tag='sp-results') \
11
AS \
12
SELECT country, AVG(num) FROM STREAM:samples \
13
WINDOW TUMBLING (1 SECOND) \
14
WHERE country='Chile' \
15
GROUP BY country;" \
16
-o stdout -m 'sp-results' -f 1
Copied!
Last modified 4mo ago
Export as PDF
Copy link