Azure output plugin allows to ingest your records into Azure Log Analytics service.

To get more details about how to setup Azure Log Analytics, please refer to the following documentation: Azure Log Analytics

Configuration Parameters

Getting Started

In order to insert records into an Azure Log Analytics instance, you can run the plugin from the command line or through the configuration file:

Command Line

The azure plugin, can read the parameters from the command line in two ways, through the -p argument (property), e.g:

Configuration File

In your main configuration file append the following Input & Output sections:

Counter is a very simple plugin that counts how many records it's getting upon flush time. Plugin output is as follows:

[TIMESTAMP, NUMBER_OF_RECORDS_NOW] (total = RECORDS_SINCE_IT_STARTED)

Getting Started

You can run the plugin from the command line or through the configuration file:

Command Line

From the command line you can let Fluent Bit count up a data with the following options:

Configuration File

In your main configuration file append the following Input & Output sections:

Testing

Once Fluent Bit is running, you will see the reports in the output interface similar to this:

BigQuery output plugin is an experimental plugin that allows you to stream records into Google Cloud BigQuery service. The implementation does not support the following, which would be expected in a full production version:

• Application Default Credentials.

• Data deduplication using insertId.

• using templateSuffix.

Google Cloud Configuration

Fluent Bit streams data into an existing BigQuery table using a service account that you specify. Therefore, before using the BigQuery output plugin, you must create a service account, create a BigQuery dataset and table, authorize the service account to write to the table, and provide the service account credentials to Fluent Bit.

Creating a Service Account

To stream data into BigQuery, the first step is to create a Google Cloud service account for Fluent Bit:

Creating a BigQuery Dataset and Table

Fluent Bit does not create datasets or tables for your data, so you must create these ahead of time. You must also grant the service account WRITER permission on the dataset:

Within the dataset you will need to create a table for the data to reside in. You can follow the following instructions for creating your table. Pay close attention to the schema. It must match the schema of your output JSON. Unfortunately, since BigQuery does not allow dots in field names, you will need to use a filter to change the fields for many of the standard inputs (e.g, mem or cpu).

Retrieving Service Account Credentials

Fluent Bit BigQuery output plugin uses a JSON credentials file for authentication credentials. Download the credentials file by following these instructions:

Configurations Parameters

Configuration File

If you are using a Google Cloud Credentials File, the following configuration is enough to get you started:

The Amazon Kinesis Data Firehose output plugin allows to ingest your records into the service.

This is the documentation for the core Fluent Bit Firehose plugin written in C. It can replace the Golang Fluent Bit plugin released last year. The Golang plugin was named firehose; this new high performance and highly efficient firehose plugin is called kinesis_firehose to prevent conflicts/confusion.

FlowCounter is the protocol to count records. The flowcounter output plugin allows to count up records and its size.

Configuration Parameters

The plugin supports the following configuration parameters:

The Slack output plugin delivers records or messages to your preferred Slack channel. It formats the outgoing content in JSON format for readability.

This connector uses the Slack Incoming Webhooks feature to post messages to Slack channels. Using this plugin in conjunction with the Stream Processor is a good combination for alerting.

Slack Webhook

Before to configure this plugin, make sure to setup your Incoming Webhook, for a detailed step-by-step instruction review the following official document:

The Datadog output plugin allows to ingest your logs into Datadog.

Before you begin, you need a Datadog account, a Datadog API key, and you need to activate Datadog Logs Management.

Configuration Parameters

Configuration File

Get started quickly with this configuration file:

Troubleshooting

403 Forbidden

If you get a 403 Forbidden error response, double check that you have a valid and that you have .

The Amazon CloudWatch output plugin allows to ingest your records into the CloudWatch Logs service. Support for CloudWatch Metrics is also provided via EMF.

This is the documentation for the core Fluent Bit CloudWatch plugin written in C. It can replace the aws/amazon-cloudwatch-logs-for-fluent-bit Golang Fluent Bit plugin released last year. The Golang plugin was named cloudwatch; this new high performance CloudWatch plugin is called cloudwatch_logs to prevent conflicts/confusion. Check the amazon repo for the Golang plugin for details on the deprecation/migration plan for the original plugin.

Configuration Parameters

Getting Started

In order to send records into Amazon Cloudwatch, you can run the plugin from the command line or through the configuration file:

Command Line

The cloudwatch plugin, can read the parameters from the command line through the -p argument (property), e.g:

Configuration File

In your main configuration file append the following Output section:

Metrics Tutorial

Fluent Bit has different input plugins (cpu, mem, disk, netif) to collect host resource usage metrics. cloudwatch_logs output plugin can be used to send these host metrics to CloudWatch in Embedded Metric Format (EMF). If data comes from any of the above mentioned input plugins, cloudwatch_logs output plugin will convert them to EMF format and sent to CloudWatch as JSON log. Additionally, if we set json/emf as the value of log_format config option, CloudWatch will extract custom metrics from embedded JSON payload.

Note: Right now, only cpu and mem metrics can be sent to CloudWatch.

For using the mem input plugin and sending memory usage metrics to CloudWatch, we can consider the following example config file. Here, we use the aws filter which adds ec2_instance_id and az (availability zone) to the log records. Later, in the output config section, we set ec2_instance_id as our metric dimension.

The following config will set two dimensions to all of our metrics- ec2_instance_id and az.

AWS for Fluent Bit

Amazon distributes a container image with Fluent Bit and these plugins.

GitHub

Amazon ECR Public Gallery

Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command:

For example, you can pull the image with latest version by:

If you see errors for image pull limits, try log into public ECR with your AWS credentials:

You can check the for more details

Docker Hub

Amazon ECR

You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:

For more see .

The null output plugin just throws away events.

Configuration Parameters

The plugin doesn't support configuration parameters.

Getting Started

You can run the plugin from the command line or through the configuration file:

Command Line

From the command line you can let Fluent Bit throws away events with the following options:

Configuration File

In your main configuration file append the following Input & Output sections:

The kafka-rest output plugin, allows to flush your records into a server. The following instructions assumes that you have a fully operational Kafka REST Proxy and Kafka services running in your environment.

Configuration Parameters

The file output plugin allows to write the data received through the input plugin to file.

Configuration Parameters

The plugin supports the following configuration parameters:

Kafka output plugin allows to ingest your records into an service. This plugin use the official (built-in dependency)

Configuration Parameters

The Azure Blob output plugin allows ingesting your records into service. This connector is designed to use the Append Blob and Block Blob API.

Our plugin works with the official Azure Service and also can be configured to be used with a service emulator such as .

Azure Storage Account

Before getting started, make sure you already have an Azure Storage account. As a reference, the following link explains step-by-step how to set up your account:

The stdout output plugin allows to print to the standard output the data received through the input plugin. Their usage is very simple as follows:

Configuration Parameters

LogDNA is an intuitive cloud based log management system that provides you an easy interface to query your logs once they are stored.

The Fluent Bit logdna output plugin allows you to send your log or events to a LogDNA compliant service like:

• LogDNA

Before to get started with the plugin configuration, make sure to obtain the proper account to get access to the service. You can start with a free trial in the following link:

Configuration Parameters

Auto Enrichment & Data Discovery

One of the features of Fluent Bit + LogDNA integration is the ability to auto enrich each record with further context.

When the plugin process each record (or log), it tries to lookup for specific key names that might contain specific context for the record in question, the following table describe the keys and the discovery logic:

Getting Started

The following configuration example, will emit a dummy example record and ingest it on LogDNA. Copy and paste the following content in a file called logdna.conf:

run Fluent Bit with the new configuration file:

Fluent Bit output:

Your record will be available and visible in your LogDNA dashboard after a few seconds.

Query your Data in LogDNA

In your LogDNA dashboard, go to the top filters and mark the Tags aa and bb, then you will be able to see your records as the example below:

The nats output plugin, allows to flush your records into a NATS Server end point. The following instructions assumes that you have a fully operational NATS Server in place.

In order to flush records, the nats plugin requires to know two parameters:

In order to override the default configuration values, the plugin uses the optional Fluent Bit network address format, e.g:

Running

only requires to know that it needs to use the nats output plugin, if no extra information is given, it will use the default values specified in the above table.

As described above, the target service and storage point can be changed, e.g:

Data format

For every set of records flushed to a NATS Server, Fluent Bit uses the following JSON format:

Each record is an individual entity represented in a JSON array that contains a UNIX_TIMESTAMP and a JSON map with a set of key/values. A summarized output of the CPU input plugin will looks as this:

The td output plugin, allows to flush your records into the Treasure Data cloud service.

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

In order to start inserting records into , you can run the plugin from the command line or through the configuration file:

Command Line:

Ideally you don't want to expose your API key from the command line, using a configuration file is higly desired.

Configuration File

In your main configuration file append the following Input & Output sections:

The Amazon S3 output plugin allows you to ingest your records into the cloud object store.

The plugin can upload data to S3 using the or using S3 . Multipart is the default and is recommended; Fluent Bit will stream data in a series of 'parts'. This limits the amount of data it has to buffer on disk at any point in time. By default, every time 5 MiB of data have been received, a new 'part' will be uploaded. The plugin can create files up to gigabytes in size from many small chunks/parts using the multipart API. All aspects of the upload process are configurable using the configuration options.

The plugin allows you to specify a maximum file size, and a timeout for uploads. A file will be created in S3 when the max size is reached, or the timeout is reached- whichever comes first.

Records are stored in files in S3 as newline delimited JSON.

The Syslog output plugin allows you to deliver messages to Syslog servers, it supports RFC3164 and RFC5424 formats through different transports such as UDP, TCP or TLS.

As of Fluent Bit v1.5.3, the configuration is very strict in terms that you must be aware about the structure of your original record, so you can configure the plugin to use specific keys to compose your outgoing Syslog message.

Future versions of Fluent Bit are expanding this plugin feature set to support better handling of keys and message composing.

The influxdb output plugin, allows to flush your records into a InfluxDB time series database. The following instructions assumes that you have a fully operational InfluxDB service running in your system.

Configuration Parameters

TLS / SSL

InfluxDB output plugin supports TTL/SSL, for more details about the properties available and general configuration, please refer to the section.

Getting Started

In order to start inserting records into an InfluxDB service, you can run the plugin from the command line or through the configuration file:

Command Line

The influxdb plugin, can read the parameters from the command line in two ways, through the -p argument (property) or setting them directly through the service URI. The URI format is the following:

Using the format specified, you could start Fluent Bit through:

Configuration File

In your main configuration file append the following Input & Output sections:

Tagging

Basic example of Tag_Keys usage:

With Auto_Tags=On in this example cause error, because every parsed field value type is string. Best usage of this option in metrics like record where one ore more field value is not string typed.

Testing

Before to start Fluent Bit, make sure the target database exists on InfluxDB, using the above example, we will insert the data into a fluentbit database.

1. Create database

Log into InfluxDB console:

Create the database:

Check the database exists:

2. Run Fluent Bit

The following command will gather CPU metrics from the system and send the data to InfluxDB database every five seconds:

Note that all records coming from the cpu input plugin, have a tag cpu, this tag is used to generate the measurement in InfluxDB

3. Query the data

From InfluxDB console, choose your database:

Now query some specific fields:

The CPU input plugin gather more metrics per CPU core, in the above example we just selected three specific metrics. The following query will give a full result:

4. View tags

Query tagged keys:

And now query method key values:

GELF is Graylog Extended Log Format. The GELF output plugin allows to send logs in GELF format directly to a Graylog input using TLS, TCP or UDP protocols.

The following instructions assumes that you have a fully operational Graylog server running in your environment.

Configuration Parameters

According to GELF Payload Specification, there are some mandatory and optional fields which are used by Graylog in GELF format. These fields are determined with Gelf\*_Key_ key in this plugin.

TLS / SSL

GELF output plugin supports TLS/SSL, for more details about the properties available and general configuration, please refer to the section.

Notes

• If you're using Fluent Bit to collect Docker logs, note that Docker places your log in JSON under key log. So you can set log as your Gelf_Short_Message_Key to send everything in Docker logs to Graylog. In this case, you need your log value to be a string; so don't parse it using JSON parser.

• The order of looking up the timestamp in this plugin is as follows:

Configuration File Example

If you're using Fluent Bit for shipping Kubernetes logs, you can use something like this as your configuration file:

By default, GELF tcp uses port 12201 and Docker places your logs in /var/log/containers directory. The logs are placed in value of the log key. For example, this is a log saved by Docker:

If you use and use a Parser like the docker parser shown above, it decodes your message and extracts data (and any other present) field. This is how this log in looks like after decoding:

Now, this is what happens to this log:

1. Fluent Bit GELF plugin adds "version": "1.1" to it.

2. The , unnests fields inside log key. In our example, it puts data alongside stream and time.

Finally, this is what our Graylog server input sees:

Splunk output plugin allows to ingest your records into a Splunk Enterprise service through the HTTP Event Collector (HEC) interface.

To get more details about how to setup the HEC in Splunk please refer to the following documentation: Splunk / Use the HTTP Event Collector

Configuration Parameters

TLS / SSL

Splunk output plugin supports TTL/SSL, for more details about the properties available and general configuration, please refer to the section.

Getting Started

In order to insert records into a Splunk service, you can run the plugin from the command line or through the configuration file:

Command Line

The splunk plugin, can read the parameters from the command line in two ways, through the -p argument (property), e.g:

Configuration File

In your main configuration file append the following Input & Output sections:

Data format

By default, the Splunk output plugin nests the record under the event key in the payload sent to the HEC. It will also append the time of the record to a top level time key.

If you would like to customize any of the Splunk event metadata, such as the host or target index, you can set Splunk_Send_Raw On in the plugin configuration, and add the metadata as keys/values in the record. Note: with Splunk_Send_Raw enabled, you are responsible for creating and populating the event section of the payload.

For example, to add a custom index and hostname:

This will create a payload that looks like:

For more information on the Splunk HEC payload format and all event meatadata Splunk accepts, see here:

Sending Raw Events

If the option splunk_send_raw has been enabled, the user must take care to put all log details in the event field, and only specify fields known to Splunk in the top level event, if there is a mismatch, Splunk will return a HTTP error 400.

Consider the following example:

splunk_send_raw off

splunk_send_raw on

For up to date information about the valid keys in the top level object, refer to the Splunk documentation:

New Relic is a data management platform that gives you real-time insights of your data for developers, operations and management teams.

The Fluent Bit nrlogs output plugin allows you to send your logs to New Relic service.

Before to get started with the plugin configuration, make sure to obtain the proper account to get access to the service. You can register and start with a free trial in the following link:

• New Relic Sign Up

Configuration Parameters

The following configuration example, will emit a dummy example record and ingest it on New Relic. Copy and paste the following content in a file called newrelic.conf:

run Fluent Bit with the new configuration file:

Fluent Bit output:

The tcp output plugin allows to send records to a remote TCP server. The payload can be formatted in different ways as required.

Configuration Parameters

TLS Configuration Parameters

The following parameters are available to configure a secure channel connection through TLS:

Command Line

We have specified to gather usage metrics and send them in JSON lines mode to a remote end-point using netcat service, e.g:

Start the TCP listener

Run the following in a separate terminal, netcat will start listening for messages on TCP port 5170

Start Fluent Bit

No more, no less, it just works.

Stackdriver output plugin allows to ingest your records into Google Cloud Stackdriver Logging service.

Before to get started with the plugin configuration, make sure to obtain the proper credentials to get access to the service. We strongly recommend to use a common JSON credentials file, reference link:

• Creating a Google Service Account for Stackdriver

Your goal is to obtain a credentials JSON file that will be used later by Fluent Bit Stackdriver output plugin.

Configuration Parameters

Configuration File

If you are using a Google Cloud Credentials File, the following configuration is enough to get started:

Example configuration file for k8s resource type:

local_resource_id is used by stackdriver output plugin to set the labels field for different k8s resource types. Stackdriver plugin will try to find the local_resource_id field in the log entey. If there is no field logging.googleapis.com/local_resource_id in the log, the plugin will then construct it by using the tag value of the log.

The local_resource_id should be in format:

• k8s_container.<namespace_name>.<pod_name>.<container_name>

• k8s_node.<node_name>

• k8s_pod.<namespace_name>.<pod_name>

This implies that if there is no local_resource_id in the log entry then the tag of logs should match this format. Note that we have an option tag_prefix so it is not mandatory to use k8s_container(node/pod) as the prefix for tag.

Troubleshooting Notes

Upstream connection error

Github reference:

An upstream connection error means Fluent Bit was not able to reach Google services, the error looks like this:

This belongs to a network issue by the environment where Fluent Bit is running, make sure that from the Host, Container or Pod you can reach the following Google end-points:

Fail to process local_resource_id

The error looks like this:

Do following check:

• If the log entry does not contain the local_resource_id field, does the tag of the log match for format?

• If tag_prefix is configured, does the prefix of tag specified in the input plugin match the tag_prefix?

  Other implementations

Stackdriver officially supports a .

We plan to support some . Use cases of special fields is .

The http output plugin allows to flush your records into a HTTP endpoint. For now the functionality is pretty basic and it issues a POST request with the data records in (or JSON) format.

Configuration Parameters

Loki is multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate.

The Fluent Bit loki built-in output plugin allows you to send your log or events to a Loki service. It support data enrichment with Kubernetes labels, custom label keys and Tenant ID within others.

Configuration Parameters

Labels

Loki store the record logs inside Streams, a stream is defined by a set of labels, at least one label is required.

Fluent Bit implements a flexible mechanism to set labels by using fixed key/value pairs of text but also allowing to set as labels certain keys that exists as part of the records that are being processed. Consider the following JSON record (pretty printed for readability):

If you decide that your Loki Stream will be composed by two labels called job and the value of the record key called stream , your labels configuration properties might look as follows:

As you can see the label job has the value fluentbit and the second label is configured to access the nested map called sub targeting the value of the key stream . Note that the second label name must starts with a $, that means that's a pattern so it provide you the ability to retrieve values from nested maps by using the key names.

When processing above's configuration, internally the ending labels for the stream in question becomes:

Another feature of Labels management is the ability to provide custom key names, using the same record accessor pattern we can specify the key name manually and let the value to be populated automatically at runtime, e.g:

When processing that new configuration, the internal labels will be:

Using the label_keys property

The additional configuration property called label_keys allow to specify multiple record keys that needs to be placed as part of the outgoing Stream Labels, yes, this is a similar feature than the one explained above in the labels property. Consider this as another way to set a record key in the Stream, but with the limitation that you cannot use a custom name for the key value.

The following configuration examples generate the same Stream Labels:

the above configuration accomplish the same than this one:

both will generate the following Streams label:

Kubernetes & Labels

Note that if you are running in a Kubernetes environment, you might want to enable the option auto_kubernetes_labels which will auto-populate the streams with the Pod labels for you. Consider the following configuration:

Based in the JSON example provided above, the internal stream labels will be:

Networking and TLS Configuration

This plugin inherit core Fluent Bit features to customize the network behavior and optionally enable TLS in the communication channel. For more details about the specific options available refer to the following articles:

• : timeouts, keepalive and source address

• : all about TLS configuration and certificates

Note that all options mentioned in the articles above must be enabled in the plugin configuration in question.

Getting Started

The following configuration example, will emit a dummy example record and ingest it on Loki . Copy and paste the following content in a file called out_loki.conf:

run Fluent Bit with the new configuration file:

Fluent Bit output: