Forward
Forward is the protocol used by Fluentd to route messages between peers. The forward output plugin provides interoperability between Fluent Bit and Fluentd. There are no configuration steps required besides specifying where Fluentd is located, which can be a local or a remote destination.
This plugin offers two different transports and modes:
- Forward (TCP): It uses a plain TCP connection.
- Secure Forward (TLS): when TLS is enabled, the plugin switch to Secure Forward mode.
Configuration Parameters
The following parameters are mandatory for either Forward for Secure Forward modes:
Key
Description
Default
Host
Target host where Fluent-Bit or Fluentd are listening for Forward messages.
127.0.0.1
Port
TCP Port of the target service.
24224
Time_as_Integer
Set timestamps in integer format, it enable compatibility mode for Fluentd v0.12 series.
False
Upstream
If Forward will connect to an Upstream instead of a simple host, this property defines the absolute path for the Upstream configuration file, for more details about this refer to the Upstream Servers documentation section.
​
Tag
Overwrite the tag as we transmit. This allows the receiving pipeline start fresh, or to attribute source.
​
Send_options
Always send options (with "size"=count of messages)
False
Require_ack_response
Send "chunk"-option and wait for "ack" response from server. Enables at-least-once and receiving server can control rate of traffic. (Requires Fluentd v0.14.0+ server)
False
Compress
Set to "gzip" to enable gzip compression. Incompatible with Time_as_Integer=True and tags set dynamically using the Rewrite Tag filter. (Requires Fluentd v0.14.7+ server)
​
Workers
Enables dedicated thread(s) for this output. Default value is set since version 1.8.13. For previous versions is 0.
2
Secure Forward Mode Configuration Parameters
When using Secure Forward mode, the TLS mode requires to be enabled. The following additional configuration parameters are available:
Key
Description
Default
Shared_Key
A key string known by the remote Fluentd used for authorization.
​
Empty_Shared_Key
Use this option to connect to Fluentd with a zero-length secret.
False
Username
Specify the username to present to a Fluentd server that enables
user_auth.​
Password
Specify the password corresponding to the username.
​
Self_Hostname
Default value of the auto-generated certificate common name (CN).
localhost
tls
Enable or disable TLS support
Off
tls.verify
Force certificate validation
On
tls.debug
Set TLS debug verbosity level. It accept the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational) and 4 Verbose
1
tls.ca_file
Absolute path to CA certificate file
​
tls.crt_file
Absolute path to Certificate file.
​
tls.key_file
Absolute path to private Key file.
​
tls.key_passwd
Optional password for tls.key_file file.
​
Forward Setup
Before proceeding, make sure that Fluentd is installed, if it's not the case please refer to the following Fluentd Installation document and go ahead with that.
Once Fluentd is installed, create the following configuration file example that will allow us to stream data into it:
1
<source>
2
type forward
3
bind 0.0.0.0
4
port 24224
5
</source>
6
​
7
<match fluent_bit>
8
type stdout
9
</match>
Copied!
That configuration file specifies that it will listen for TCP connections on the port 24224 through the forward input type. Then for every message with a fluent_bit TAG, will print the message to the standard output.
1
$ fluentd -c test.conf
2
2017-03-23 11:50:43 -0600 [info]: reading config file path="test.conf"
3
2017-03-23 11:50:43 -0600 [info]: starting fluentd-0.12.33
4
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.1'
5
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-docker' version '0.1.0'
6
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-elasticsearch' version '1.4.0'
7
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-flatten-hash' version '0.2.0'
8
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.0.4'
9
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-influxdb' version '0.2.8'
10
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-json-in-json' version '0.1.4'
11
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-mongo' version '0.7.10'
12
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-out-http' version '0.1.3'
13
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-parser' version '0.6.0'
14
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-record-reformer' version '0.7.0'
15
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.1'
16
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-stdin' version '0.1.1'
17
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-td' version '0.10.27'
18
2017-03-23 11:50:43 -0600 [info]: adding match pattern="fluent_bit" type="stdout"
19
2017-03-23 11:50:43 -0600 [info]: adding source type="forward"
20
2017-03-23 11:50:43 -0600 [info]: using configuration file: <ROOT>
21
<source>
22
type forward
23
bind 0.0.0.0
24
port 24224
25
</source>
26
<match fluent_bit>
27
type stdout
28
</match>
29
</ROOT>
30
2017-03-23 11:50:43 -0600 [info]: listening fluent socket on 0.0.0.0:24224
Copied!
Fluent Bit + Forward Setup
Now that Fluentd is ready to receive messages, we need to specify where the forward output plugin will flush the information using the following format:
1
bin/fluent-bit -i INPUT -o forward://HOST:PORT
Copied!
If the TAG parameter is not set, the plugin will retain the tag. Keep in mind that TAG is important for routing rules inside Fluentd.
1
$ bin/fluent-bit -i cpu -t fluent_bit -o forward://127.0.0.1:24224
Copied!
1
2017-03-23 11:53:06 -0600 fluent_bit: {"cpu_p":0.0,"user_p":0.0,"system_p":0.0,"cpu0.p_cpu":0.0,"cpu0.p_user":0.0,"cpu0.p_system":0.0,"cpu1.p_cpu":0.0,"cpu1.p_user":0.0,"cpu1.p_system":0.0,"cpu2.p_cpu":0.0,"cpu2.p_user":0.0,"cpu2.p_system":0.0,"cpu3.p_cpu":1.0,"cpu3.p_user":1.0,"cpu3.p_system":0.0}
2
2017-03-23 11:53:07 -0600 fluent_bit: {"cpu_p":2.25,"user_p":2.0,"system_p":0.25,"cpu0.p_cpu":3.0,"cpu0.p_user":3.0,"cpu0.p_system":0.0,"cpu1.p_cpu":1.0,"cpu1.p_user":1.0,"cpu1.p_system":0.0,"cpu2.p_cpu":1.0,"cpu2.p_user":1.0,"cpu2.p_system":0.0,"cpu3.p_cpu":3.0,"cpu3.p_user":2.0,"cpu3.p_system":1.0}
3
2017-03-23 11:53:08 -0600 fluent_bit: {"cpu_p":1.75,"user_p":1.0,"system_p":0.75,"cpu0.p_cpu":2.0,"cpu0.p_user":1.0,"cpu0.p_system":1.0,"cpu1.p_cpu":3.0,"cpu1.p_user":1.0,"cpu1.p_system":2.0,"cpu2.p_cpu":3.0,"cpu2.p_user":2.0,"cpu2.p_system":1.0,"cpu3.p_cpu":2.0,"cpu3.p_user":1.0,"cpu3.p_system":1.0}
4
2017-03-23 11:53:09 -0600 fluent_bit: {"cpu_p":4.75,"user_p":3.5,"system_p":1.25,"cpu0.p_cpu":4.0,"cpu0.p_user":3.0,"cpu0.p_system":1.0,"cpu1.p_cpu":5.0,"cpu1.p_user":4.0,"cpu1.p_system":1.0,"cpu2.p_cpu":3.0,"cpu2.p_user":2.0,"cpu2.p_system":1.0,"cpu3.p_cpu":5.0,"cpu3.p_user":4.0,"cpu3.p_system":1.0}
Copied!
Fluent Bit + Secure Forward Setup
DISCLAIMER: the following example does not consider the generation of certificates for best practice on production environments.
Secure Forward aims to provide a secure channel of communication with the remote Fluentd service using TLS.
Fluent Bit
Paste this content in a file called flb.conf:
1
[SERVICE]
2
Flush 5
3
Daemon off
4
Log_Level info
5
​
6
[INPUT]
7
Name cpu
8
Tag cpu_usage
9
​
10
[OUTPUT]
11
Name forward
12
Match *
13
Host 127.0.0.1
14
Port 24284
15
Shared_Key secret
16
Self_Hostname flb.local
17
tls on
18
tls.verify off
Copied!
Fluentd
Paste this content in a file called fld.conf:
1
<source>
2
@type secure_forward
3
self_hostname myserver.local
4
shared_key secret
5
secure no
6
</source>
7
​
8
<match **>
9
@type stdout
10
</match>
Copied!
If you're using Fluentd v1, set up it as below:
1
<source>
2
@type forward
3
<transport tls>
4
cert_path /etc/td-agent/certs/fluentd.crt
5
private_key_path /etc/td-agent/certs/fluentd.key
6
private_key_passphrase password
7
</transport>
8
<security>
9
self_hostname myserver.local
10
shared_key secret
11
</security>
12
</source>
13
​
14
<match **>
15
@type stdout
16
</match>
Copied!
Test Communication
Start Fluentd:
1
$ fluentd -c fld.conf
Copied!
Start Fluent Bit:
1
$ fluent-bit -c flb.conf
Copied!
After five seconds, Fluent Bit will write records to Fluentd. In Fluentd output you will see a message like this:
1
2017-03-23 13:34:40 -0600 [info]: using configuration file: <ROOT>
2
<source>
3
@type secure_forward
4
self_hostname myserver.local
5
shared_key xxxxxx
6
secure no
7
</source>
8
<match **>
9
@type stdout
10
</match>
11
</ROOT>
12
2017-03-23 13:34:41 -0600 cpu_usage: {"cpu_p":1.0,"user_p":0.75,"system_p":0.25,"cpu0.p_cpu":1.0,"cpu0.p_user":1.0,"cpu0.p_system":0.0,"cpu1.p_cpu":2.0,"cpu1.p_user":1.0,"cpu1.p_system":1.0,"cpu2.p_cpu":1.0,"cpu2.p_user":1.0,"cpu2.p_system":0.0,"cpu3.p_cpu":2.0,"cpu3.p_user":1.0,"cpu3.p_system":1.0}
13
2017-03-23 13:34:42 -0600 cpu_usage: {"cpu_p":1.75,"user_p":1.75,"system_p":0.0,"cpu0.p_cpu":3.0,"cpu0.p_user":3.0,"cpu0.p_system":0.0,"cpu1.p_cpu":2.0,"cpu1.p_user":2.0,"cpu1.p_system":0.0,"cpu2.p_cpu":0.0,"cpu2.p_user":0.0,"cpu2.p_system":0.0,"cpu3.p_cpu":1.0,"cpu3.p_user":1.0,"cpu3.p_system":0.0}
14
2017-03-23 13:34:43 -0600 cpu_usage: {"cpu_p":1.75,"user_p":1.25,"system_p":0.5,"cpu0.p_cpu":3.0,"cpu0.p_user":3.0,"cpu0.p_system":0.0,"cpu1.p_cpu":2.0,"cpu1.p_user":2.0,"cpu1.p_system":0.0,"cpu2.p_cpu":0.0,"cpu2.p_user":0.0,"cpu2.p_system":0.0,"cpu3.p_cpu":1.0,"cpu3.p_user":0.0,"cpu3.p_system":1.0}
15
2017-03-23 13:34:44 -0600 cpu_usage: {"cpu_p":5.0,"user_p":3.25,"system_p":1.75,"cpu0.p_cpu":4.0,"cpu0.p_user":2.0,"cpu0.p_system":2.0,"cpu1.p_cpu":8.0,"cpu1.p_user":5.0,"cpu1.p_system":3.0,"cpu2.p_cpu":4.0,"cpu2.p_user":3.0,"cpu2.p_system":1.0,"cpu3.p_cpu":4.0,"cpu3.p_user":2.0,"cpu3.p_system":2.0}
Copied!
Last modified 2mo ago