Forward
Forward is the protocol used by Fluentd to route messages between peers. The forward output plugin allows to provide interoperability between Fluent Bit and Fluentd. There are not configuration steps required besides to specify where Fluentd is located, it can be in the local host or a in a remote machine.
This plugin offers two different transports and modes:
Forward (TCP): It uses a plain TCP connection.
Secure Forward (TLS): when TLS is enabled, the plugin switch to Secure Forward mode.
Configuration Parameters
The following parameters are mandatory for either Forward for Secure Forward modes:
Key | Description | Default |
Host | Target host where Fluent-Bit or Fluentd are listening for Forward messages. | 127.0.0.1 |
Port | TCP Port of the target service. | 24224 |
Time_as_Integer | Set timestamps in integer format, it enable compatibility mode for Fluentd v0.12 series. | False |
Upstream | If Forward will connect to an Upstream instead of a simple host, this property defines the absolute path for the Upstream configuration file, for more details about this refer to the Upstream Servers documentation section. | |
Tag | Overwrite the tag as we transmit. This allows the receiving pipeline start fresh, or to attribute source. | |
Send_options | Always send options (with "size"=count of messages) | False |
Require_ack_response | Send "chunk"-option and wait for "ack" response from server. Enables at-least-once and receiving server can control rate of traffic. (Requires Fluentd v0.14.0+ server) | False |
Compress | Set to "gzip" to enable gzip compression. Incompatible with Time_as_Integer=True and tags set dynamically using the Rewrite Tag filter. (Requires Fluentd v0.14.7+ server) |
Secure Forward Mode Configuration Parameters
When using Secure Forward mode, the TLS mode requires to be enabled. The following additional configuration parameters are available:
Key | Description | Default |
Shared_Key | A key string known by the remote Fluentd used for authorization. | |
Empty_Shared_Key | Use this option to connect to Fluentd with a zero-length secret. | False |
Username | Specify the username to present to a Fluentd server that enables | |
Password | Specify the password corresponding to the username. | |
Self_Hostname | Default value of the auto-generated certificate common name (CN). | localhost |
tls | Enable or disable TLS support | Off |
tls.verify | Force certificate validation | On |
tls.debug | Set TLS debug verbosity level. It accept the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational) and 4 Verbose | 1 |
tls.ca_file | Absolute path to CA certificate file | |
tls.crt_file | Absolute path to Certificate file. | |
tls.key_file | Absolute path to private Key file. | |
tls.key_passwd | Optional password for tls.key_file file. |
Forward Setup
Before proceeding, make sure that Fluentd is installed in your system, if it's not the case please refer to the following Fluentd Installation document and go ahead with that.
Once Fluentd is installed, create the following configuration file example that will allow us to stream data into it:
That configuration file specifies that it will listen for TCP connections on the port 24224 through the forward input type. Then for every message with a fluent_bit TAG, will print the message to the standard output.
In one terminal launch Fluentd specifying the new configuration file created (in_fluent-bit.conf):
Fluent Bit + Forward Setup
Now that Fluentd is ready to receive messages, we need to specify where the forward output plugin will flush the information using the following format:
If the TAG parameter is not set, the plugin will retain the tag. Keep in mind that TAG is important for routing rules inside Fluentd.
Using the CPU input plugin as an example we will flush CPU metrics to Fluentd with tag fluent_bit:
Now on the Fluentd side, you will see the CPU metrics gathered in the last seconds:
So we gathered CPU metrics and flushed them out to Fluentd properly.
Fluent Bit + Secure Forward Setup
DISCLAIMER: the following example do not consider the generation of certificates for a proper usage of production environments.
Secure Forward aims to provide a secure channel of communication with the remote Fluentd service using TLS. Above there is a minimalist configuration for testing purposes.
Fluent Bit
Paste this content in a file called flb.conf:
Fluentd
Paste this content in a file called fld.conf:
If you're using Fluentd v1, set up it as below:
Test Communication
Start Fluentd:
Start Fluent Bit:
After five seconds, Fluent Bit will write the records to Fluentd. In Fluentd output you will see a message like this:
Last updated