Throttle
The Throttle Filter plugin sets the average Rate of messages per Interval, based on leaky bucket and sliding window algorithm. In case of overflood, it will leak within certain rate.

Configuration Parameters

The plugin supports the following configuration parameters:
Key
Value Format
Description
Rate
Integer
Amount of messages for the time.
Window
Integer
Amount of intervals to calculate average over. Default 5.
Interval
String
Time interval, expressed in "sleep" format. e.g 3s, 1.5m, 0.5h etc
Print_Status
Bool
Whether to print status messages with current rate and the limits to information logs

Functional description

Lets imagine we have configured:
1
Rate 5
2
Window 5
3
Interval 1s
Copied!
we received 1 message first second, 3 messages 2nd, and 5 3rd. As you can see, disregard that Window is actually 5, we use "slow" start to prevent overflooding during the startup.
1
+-------+-+-+-+
2
|1|3|5| | | | |
3
+-------+-+-+-+
4
| 3 | average = 3, and not 1.8 if you calculate 0 for last 2 panes.
5
+-----+
Copied!
But as soon as we reached Window size * Interval, we will have true sliding window with aggregation over complete window.
1
+-------------+
2
|1|3|5|7|3|4| |
3
+-------------+
4
| 4.4 |
5
----------+
Copied!
When we have average over window is more than Rate, we will start dropping messages, so that
1
+-------------+
2
|1|3|5|7|3|4|7|
3
+-------------+
4
| 5.2 |
5
+---------+
Copied!
will become:
1
+-------------+
2
|1|3|5|7|3|4|6|
3
+-------------+
4
| 5 |
5
+---------+
Copied!
As you can see, last pane of the window was overwritten and 1 message was dropped.

Interval vs Window size

You might noticed possibility to configure Interval of the Window shift. It is counter intuitive, but there is a difference between two examples above:
1
Rate 60
2
Window 5
3
Interval 1m
Copied!
and
1
Rate 1
2
Window 300
3
Interval 1s
Copied!
Even though both examples will allow maximum Rate of 60 messages per minute, first example may get all 60 messages within first second, and will drop all the rest for the entire minute:
1
XX XX XX
2
XX XX XX
3
XX XX XX
4
XX XX XX
5
XX XX XX
6
XX XX XX
7
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Copied!
While the second example will not allow more than 1 message per second every second, making output rate more smooth:
1
X X X X X X
2
XXXX XXXX XXXX XXXX XXXX XXXX
3
+-+-+-+-+-+--+-+-+-+-+-+-+-+-+-+
Copied!
It may drop some data if the rate is ragged. I would recommend to use bigger interval and rate for streams of rare but important events, while keep Window bigger and Interval small for constantly intensive inputs.

Command Line

Note: It's suggested to use a configuration file.
The following command will load the tail plugin and read the content of lines.txt file. Then the throttle filter will apply a rate limit and only pass the records which are read below the certain rate:
1
$ bin/fluent-bit -i tail -p 'path=lines.txt' -F throttle -p 'rate=1' -m '*' -o stdout
Copied!

Configuration File

1
[INPUT]
2
Name tail
3
Path lines.txt
4
5
[FILTER]
6
Name throttle
7
Match *
8
Rate 1000
9
Window 300
10
Interval 1s
11
12
[OUTPUT]
13
Name stdout
14
Match *
Copied!
The example above will pass 1000 messages per second in average over 300 seconds.
Last modified 1yr ago