1.9
Installation
Administration
Data Pipeline
Stream Processing
Upstream Servers
It's common that Fluent Bit output plugins aims to connect to external services to deliver the logs over the network, this is the case of HTTP, Elasticsearch and Forward within others. Being able to connect to one node (host) is normal and enough for more of the use cases, but there are other scenarios where balancing across different nodes is required. The Upstream feature provides such capability.
An Upstream defines a set of nodes that will be targeted by an output plugin, by the nature of the implementation an output plugin must support the Upstream feature. The following plugin(s) have Upstream support:
The current balancing mode implemented is round-robin.
Configuration
To define an Upstream it's required to create an specific configuration file that contains an UPSTREAM and one or multiple NODE sections. The following table describe the properties associated to each section. Note that all of them are mandatory:
Section
Key
Description
UPSTREAM
name
Defines a name for the Upstream in question.
NODE
name
Defines a name for the Node in question.
​
host
IP address or hostname of the target host.
​
port
TCP port of the target service.
Nodes and specific plugin configuration
A Node might contain additional configuration keys required by the plugin, on that way we provide enough flexibility for the output plugin, a common use case is Forward output where if TLS is enabled, it requires a shared key (more details in the example below).
Nodes and TLS (Transport Layer Security)
In addition to the properties defined in the table above, the network operations against a defined node can optionally be done through the use of TLS for further encryption and certificates use.
The TLS options available are described in the TLS/SSL section and can be added to the any Node section.
Configuration File Example
The following example defines an Upstream called forward-balancing which aims to be used by Forward output plugin, it register three Nodes:
- node-1: connects to 127.0.0.1:43000
- node-2: connects to 127.0.0.1:44000
- node-3: connects to 127.0.0.1:45000 using TLS without verification. It also defines a specific configuration option required by Forward output called shared_key.
1
[UPSTREAM]
2
name forward-balancing
3
​
4
[NODE]
5
name node-1
6
host 127.0.0.1
7
port 43000
8
​
9
[NODE]
10
name node-2
11
host 127.0.0.1
12
port 44000
13
​
14
[NODE]
15
name node-3
16
host 127.0.0.1
17
port 45000
18
tls on
19
tls.verify off
20
shared_key secret
Copied!
Note that every Upstream definition must exists on it own configuration file in the file system. Adding multiple Upstreams in the same file or different files is not allowed.
Last modified 3mo ago
Export as PDF
Copy link