The Amazon Kinesis Data Firehose output plugin allows to ingest your records into the Firehose service.

This is the documentation for the core Fluent Bit Firehose plugin written in C. It can replace the aws/amazon-kinesis-firehose-for-fluent-bit Golang Fluent Bit plugin released last year. The Golang plugin was named firehose; this new high performance and highly efficient firehose plugin is called kinesis_firehose to prevent conflicts/confusion.

See here for details on how AWS credentials are fetched.

Configuration Parameters

Getting Started

In order to send records into Amazon Kinesis Data Firehose, you can run the plugin from the command line or through the configuration file:

Command Line

The firehose plugin, can read the parameters from the command line through the -p argument (property), e.g:

$ fluent-bit -i cpu -o kinesis_firehose -p delivery_stream=my-stream -p region=us-west-2 -m '*' -f 1

Configuration File

In your main configuration file append the following Output section:

[OUTPUT]
    Name  kinesis_firehose
    Match *
    region us-east-1
    delivery_stream my-stream

Permissions

The following AWS IAM permissions are required to use this plugin:

{
	"Version": "2012-10-17",
	"Statement": [{
		"Effect": "Allow",
		"Action": [
			"firehose:PutRecordBatch"
		],
		"Resource": "*"
	}]
}

Worker support

Fluent Bit 1.7 adds a new feature called workers which enables outputs to have dedicated threads. This kinesis_firehose plugin fully supports workers.

Example:

[OUTPUT]
    Name  kinesis_firehose
    Match *
    region us-east-1
    delivery_stream my-stream
    workers 2

If you enable a single worker, you are enabling a dedicated thread for your Firehose output. We recommend starting with without workers, evaluating the performance, and then adding workers one at a time until you reach your desired/needed throughput. For most users, no workers or a single worker will be sufficient.

AWS for Fluent Bit

Amazon distributes a container image with Fluent Bit and these plugins.

GitHub

github.com/aws/aws-for-fluent-bit

Amazon ECR Public Gallery

aws-for-fluent-bit

Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:<tag>

For example, you can pull the image with latest version by:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

If you see errors for image pull limits, try log into public ECR with your AWS credentials:

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

You can check the Amazon ECR Public official doc for more details.

Docker Hub

amazon/aws-for-fluent-bit

Amazon ECR

You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:

aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/

For more see the AWS for Fluent Bit github repo.

The Amazon CloudWatch output plugin allows to ingest your records into the service. Support for CloudWatch Metrics is also provided via .

This is the documentation for the core Fluent Bit CloudWatch plugin written in C. It can replace the Golang Fluent Bit plugin released last year. The Golang plugin was named cloudwatch; this new high performance CloudWatch plugin is called cloudwatch_logs to prevent conflicts/confusion. Check the amazon repo for the Golang plugin for details on the deprecation/migration plan for the original plugin.

See for details on how AWS credentials are fetched.

Configuration Parameters

Getting Started

In order to send records into Amazon Cloudwatch, you can run the plugin from the command line or through the configuration file:

Command Line

The cloudwatch plugin, can read the parameters from the command line through the -p argument (property), e.g:

Configuration File

In your main configuration file append the following Output section:

Intergration with Localstack (Cloudwatch Logs)

For an instance of Localstack running at http://localhost:4566, the following configuration needs to be added to the [OUTPUT] section:

Any testing credentials can be exported as local variables, such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Permissions

The following AWS IAM permissions are required to use this plugin:

Log Stream and Group Name templating using record_accessor syntax

Here is an example usage, for a common use case- templating log group and stream names based on Kubernetes metadata.

Recall that the kubernetes filter can add metadata which will look like the following:

Using record_accessor, we can build a template based on this object.

Here is our output configuration:

With the above kubernetes metadata, the log group name will be application-logs-ip-10-1-128-166.us-east-2.compute.internal.my-namespace. And the log stream name will be myapp-5468c5d4d7-n2swr.myapp.

If the kubernetes structure is not found in the log record, then the log_group_name and log_stream_prefix will be used instead, and Fluent Bit will log an error like:

Limitations of record_accessor syntax

Notice in the example above, that the template values are separated by dot characters. This is important; the Fluent Bit record_accessor library has a limitation in the characters that can separate template variables- only dots and commas (. and ,) can come after a template variable. This is because the templating library must parse the template and determine the end of a variable.

Assume that your log records contain the metadata keys container_name and task. The following would be invalid templates because the two template variables are not separated by commas or dots:

• $task-$container_name

• $task/$container_name

• $task_$container_name

• $taskfooo$container_name

However, the following are valid:

• $task.$container_name

• $task.resource.$container_name

• $task.fooo.$container_name

And the following are valid since they only contain one template variable with nothing after it:

• fooo$task

• fooo____$task

• fooo/bar$container_name

Metrics Tutorial

Fluent Bit has different input plugins (cpu, mem, disk, netif) to collect host resource usage metrics. cloudwatch_logs output plugin can be used to send these host metrics to CloudWatch in Embedded Metric Format (EMF). If data comes from any of the above mentioned input plugins, cloudwatch_logs output plugin will convert them to EMF format and sent to CloudWatch as JSON log. Additionally, if we set json/emf as the value of log_format config option, CloudWatch will extract custom metrics from embedded JSON payload.

Note: Right now, only cpu and mem metrics can be sent to CloudWatch.

For using the mem input plugin and sending memory usage metrics to CloudWatch, we can consider the following example config file. Here, we use the aws filter which adds ec2_instance_id and az (availability zone) to the log records. Later, in the output config section, we set ec2_instance_id as our metric dimension.

The following config will set two dimensions to all of our metrics- ec2_instance_id and az.

AWS for Fluent Bit

Amazon distributes a container image with Fluent Bit and these plugins.

GitHub

Amazon ECR Public Gallery

Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command:

For example, you can pull the image with latest version by:

If you see errors for image pull limits, try log into public ECR with your AWS credentials:

Docker Hub

Amazon ECR

You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:

Counter is a very simple plugin that counts how many records it's getting upon flush time. Plugin output is as follows:

Getting Started

You can run the plugin from the command line or through the configuration file:

Command Line

From the command line you can let Fluent Bit count up a data with the following options:

Configuration File

In your main configuration file append the following Input & Output sections:

Testing

Once Fluent Bit is running, you will see the reports in the output interface similar to this:

The Azure Blob output plugin allows ingesting your records into service. This connector is designed to use the Append Blob and Block Blob API.

Our plugin works with the official Azure Service and also can be configured to be used with a service emulator such as .

Azure Storage Account

Before getting started, make sure you already have an Azure Storage account. As a reference, the following link explains step-by-step how to set up your account:

Configuration Parameters

We expose different configuration properties. The following table lists all the options available, and the next section has specific configuration details for the official service or the emulator.

Getting Started

As mentioned above, you can either deliver records to the official service or an emulator. Below we have an example for each use case.

Configuration for Azure Storage Service

The following configuration example generates a random message with a custom tag:

After you run the configuration file above, you will be able to query the data using the Azure Storage Explorer. The example above will generate the following content in the explorer:

Configuring and using Azure Emulator: Azurite

Install and run Azurite

The quickest way to get started is to install Azurite using npm:

then run the service:

Configuring Fluent Bit for Azurite

after running that Fluent Bit configuration you will see the data flowing into Azurite:

The Kusto output plugin allows to ingest your logs into an Azure Data Explorer cluster, via the Queued Ingestion mechanism. This output plugin can also be used to ingest logs into an Eventhouse cluster in Microsoft Fabric Real Time Analytics.

For ingesting into Azure Data Explorer: Creating a Kusto Cluster and Database

You can create an Azure Data Explorer cluster in one of the following ways:

• Create a free-tier cluster

• Create a fully-featured cluster

For ingesting into Microsoft Fabric Real Time Analytics : Creating an Eventhouse Cluster and KQL Database

You can create an Eventhouse cluster and a KQL database follow the following steps:

• Create an Eventhouse cluster

• Create a KQL database

Creating an Azure Registered Application

Fluent-Bit will use the application's credentials, to ingest data into your cluster.

• Register an Application

• Add a client secret

• Authorize the app in your database

Creating a Table

Fluent-Bit ingests the event data into Kusto in a JSON format, that by default will include 3 properties:

• log - the actual event payload.

• tag - the event tag.

• timestamp - the event timestamp.

A table with the expected schema must exist in order for data to be ingested properly.

.create table FluentBit (log:dynamic, tag:string, timestamp:datetime)

Optional - Creating an Ingestion Mapping

By default, Kusto will insert incoming ingestions into a table by inferring the mapped table columns, from the payload properties. However, this mapping can be customized by creatng a JSON ingestion mapping. The plugin can be configured to use an ingestion mapping via the ingestion_mapping_reference configuration key.

Configuration Parameters

Configuration File

Get started quickly with this configuration file:

[OUTPUT]
    Match *
    Name azure_kusto
    Tenant_Id <app_tenant_id>
    Client_Id <app_client_id>
    Client_Secret <app_secret>
    Ingestion_Endpoint https://ingest-<cluster>.<region>.kusto.windows.net
    Database_Name <database_name>
    Table_Name <table_name>
    Ingestion_Mapping_Reference <mapping_name>

Troubleshooting

403 Forbidden

If you get a 403 Forbidden error response, make sure that:

• You provided the correct AAD registered application credentials.

• You authorized the application to ingest into your database or table.

BigQuery output plugin is an experimental plugin that allows you to stream records into Google Cloud BigQuery service. The implementation does not support the following, which would be expected in a full production version:

• Application Default Credentials.

• Data deduplication using insertId.

• Template tables using templateSuffix.

Google Cloud Configuration

Fluent Bit streams data into an existing BigQuery table using a service account that you specify. Therefore, before using the BigQuery output plugin, you must create a service account, create a BigQuery dataset and table, authorize the service account to write to the table, and provide the service account credentials to Fluent Bit.

Creating a Service Account

To stream data into BigQuery, the first step is to create a Google Cloud service account for Fluent Bit:

• Creating a Google Cloud Service Account

Creating a BigQuery Dataset and Table

Fluent Bit does not create datasets or tables for your data, so you must create these ahead of time. You must also grant the service account WRITER permission on the dataset:

• Creating and using datasets

Within the dataset you will need to create a table for the data to reside in. You can follow the following instructions for creating your table. Pay close attention to the schema. It must match the schema of your output JSON. Unfortunately, since BigQuery does not allow dots in field names, you will need to use a filter to change the fields for many of the standard inputs (e.g, mem or cpu).

• Creating and using tables

Retrieving Service Account Credentials

Fluent Bit BigQuery output plugin uses a JSON credentials file for authentication credentials. Download the credentials file by following these instructions:

• Creating and Managing Service Account Keys

Workload Identity Federation

Using identity federation, you can grant on-premises or multi-cloud workloads access to Google Cloud resources, without using a service account key. It can be used as a more secure alternative to service account credentials. Google Cloud's workload identity federation supports several identity providers (see documentation) but Fluent Bit BigQuery plugin currently supports Amazon Web Services (AWS) only.

• Workload Identity Federation overview

You must configure workload identity federation in GCP before using it with Fluent Bit.

• Configuring workload identity federation

• Obtaining short-lived credentials with identity federation

Configurations Parameters

See Google's official documentation for further details.

Configuration File

If you are using a Google Cloud Credentials File, the following configuration is enough to get you started:

[INPUT]
    Name  dummy
    Tag   dummy

[OUTPUT]
    Name       bigquery
    Match      *
    dataset_id my_dataset
    table_id   dummy_table

The Amazon Kinesis Data Streams output plugin allows to ingest your records into the Kinesis service.

This is the documentation for the core Fluent Bit Kinesis plugin written in C. It has all the core features of the aws/amazon-kinesis-streams-for-fluent-bit Golang Fluent Bit plugin released in 2019. The Golang plugin was named kinesis; this new high performance and highly efficient kinesis plugin is called kinesis_streams to prevent conflicts/confusion.

Currently, this kinesis_streams plugin will always use a random partition key when uploading records to kinesis via the PutRecords API.

See here for details on how AWS credentials are fetched.

Configuration Parameters

Getting Started

In order to send records into Amazon Kinesis Data Streams, you can run the plugin from the command line or through the configuration file:

Command Line

The kinesis_streams plugin, can read the parameters from the command line through the -p argument (property), e.g:

$ fluent-bit -i cpu -o kinesis_streams -p stream=my-stream -p region=us-west-2 -m '*' -f 1

Configuration File

In your main configuration file append the following Output section:

[OUTPUT]
    Name  kinesis_streams
    Match *
    region us-east-1
    stream my-stream

Permissions

The following AWS IAM permissions are required to use this plugin:

{
	"Version": "2012-10-17",
	"Statement": [{
		"Effect": "Allow",
		"Action": [
			"kinesis:PutRecords"
		],
		"Resource": "*"
	}]
}

AWS for Fluent Bit

Amazon distributes a container image with Fluent Bit and these plugins.

GitHub

github.com/aws/aws-for-fluent-bit

Amazon ECR Public Gallery

aws-for-fluent-bit

Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:<tag>

For example, you can pull the image with latest version by:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

If you see errors for image pull limits, try log into public ECR with your AWS credentials:

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

You can check the Amazon ECR Public official doc for more details.

Docker Hub

amazon/aws-for-fluent-bit

Amazon ECR

You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:

aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/

For more see the AWS for Fluent Bit github repo.

The Datadog output plugin allows to ingest your logs into Datadog.

Before you begin, you need a Datadog account, a Datadog API key, and you need to activate Datadog Logs Management.

Configuration Parameters

Configuration File

Get started quickly with this configuration file:

[OUTPUT]
    Name        datadog
    Match       *
    Host        http-intake.logs.datadoghq.com
    TLS         on
    compress    gzip
    apikey      <my-datadog-api-key>
    dd_service  <my-app-service>
    dd_source   <my-app-source>
    dd_tags     team:logs,foo:bar

Troubleshooting

403 Forbidden

If you get a 403 Forbidden error response, double check that you have a valid Datadog API key and that you have activated Datadog Logs Management.

Azure output plugin allows to ingest your records into Azure Log Analytics service.

To get more details about how to setup Azure Log Analytics, please refer to the following documentation: Azure Log Analytics

Configuration Parameters

Getting Started

In order to insert records into an Azure Log Analytics instance, you can run the plugin from the command line or through the configuration file:

Command Line

The azure plugin, can read the parameters from the command line in two ways, through the -p argument (property), e.g:

$ fluent-bit -i cpu -o azure -p customer_id=abc -p shared_key=def -m '*' -f 1

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name  cpu

[OUTPUT]
    Name        azure
    Match       *
    Customer_ID abc
    Shared_Key  def

Another example using the Log_Type_Key with record-accessor, which will read the table name (or event type) dynamically from kubernetes label app, instead of Log_Type:

[INPUT]
    Name  cpu

[OUTPUT]
    Name        azure
    Match       *
    Log_Type_Key $kubernetes['labels']['app']
    Customer_ID abc
    Shared_Key  def

FlowCounter is the protocol to count records. The flowcounter output plugin allows to count up records and its size.

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

You can run the plugin from the command line or through the configuration file:

Command Line

From the command line you can let Fluent Bit count up a data with the following options:

$ fluent-bit -i cpu -o flowcounter

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name cpu
    Tag  cpu

[OUTPUT]
    Name flowcounter
    Match *
    Unit second

Testing

Once Fluent Bit is running, you will see the reports in the output interface similar to this:

$ fluent-bit -i cpu -o flowcounter
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2016/12/23 11:01:20] [ info] [engine] started
[out_flowcounter] cpu.0:[1482458540, {"counts":60, "bytes":7560, "counts/minute":1, "bytes/minute":126 }]

The es output plugin lets you ingest your records into an Elasticsearch database. To use this plugin, you must have an operational Elasticsearch service running in your environment.

Configuration Parameters

If you have used a common relational database, the parameters index and type can be compared to the database and table concepts.

TLS / SSL

Elasticsearch output plugin supports TLS/SSL. For more details about the properties available and general configuration, refer toTLS/SSL.

write_operation

The write_operation can be any of:

Get started

To insert records into an Elasticsearch service, you run the plugin from the command line or through the configuration file:

Command Line

The es plugin can read the parameters from the command line in two ways:

• Through the -p argument (property).

• Setting them directly through the service URI.

The URI format is the following:

es://host:port/index/type

Using the format specified, you could start Fluent Bit through:

fluent-bit -i cpu -t cpu -o es://192.168.2.3:9200/my_index/my_type \
    -o stdout -m '*'

Which is similar to the following command:

fluent-bit -i cpu -t cpu -o es -p Host=192.168.2.3 -p Port=9200 \
    -p Index=my_index -p Type=my_type -o stdout -m '*'

Configuration File

In your main configuration file append the following Input and Output sections.

[INPUT]
    Name  cpu
    Tag   cpu

[OUTPUT]
    Name  es
    Match *
    Host  192.168.2.3
    Port  9200
    Index my_index
    Type  my_type

About Elasticsearch field names

Some input plugins can generate messages where the field names contains dots. For Elasticsearch 2.0, this isn't allowed. The current es plugin replaces them with an underscore:

{"cpu0.p_cpu"=>17.000000}

becomes

{"cpu0_p_cpu"=>17.000000}

Use Fluent Bit ElasticSearch plugin with other services

Connect to Amazon OpenSearch or Elastic Cloud with the ElasticSearch plugin.

Amazon OpenSearch Service

The Amazon OpenSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon OpenSearch Service with IAM Authentication.

See details on how AWS credentials are fetched.

Example configuration:

[OUTPUT]
    Name  es
    Match *
    Host  vpc-test-domain-ke7thhzoo7jawsrhmm6mb7ite7y.us-west-2.es.amazonaws.com
    Port  443
    Index my_index
    Type  my_type
    AWS_Auth On
    AWS_Region us-west-2
    tls     On

Be aware that the Port is set to 443, tls is enabled, and AWS_Region is set.

Use Fluent Bit with Elastic Cloud

Fluent Bit supports connecting to Elastic Cloud by providing the cloud_id and the cloud_auth settings. cloud_auth uses the elastic user and password provided when the cluster was created. For details refer to the Cloud ID usage page.

Example configuration:

[OUTPUT]
    Name es
    Include_Tag_Key true
    Tag_Key tags
    tls On
    tls.verify Off
    Suppress_Type_Name On
    cloud_id elastic-obs-deployment:ZXVybxxxxxxxxxxxg==
    cloud_auth elastic:2vxxxxxxxxYV

In Elastic Cloud version 8 and great, the type option must be removed by setting Suppress_Type_Name On.

Without this you will see errors like:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

Troubleshooting

Use the following information to help resolve errors using the ElasticSearch plugin.

Using multiple types in a single index

Elasticsearch 6.0 can't create multiple types in a single index. An error message like the following indicates you need to update your configuration to use a single type on each index.

Rejecting mapping update to [products] as the final mapping would have more than 1 type:

This means that you can't set up your configuration like the following:.

[OUTPUT]
    Name  es
    Match foo.*
    Index search
    Type  type1

[OUTPUT]
    Name  es
    Match bar.*
    Index search
    Type  type2

For details, read the official blog post on that issue.

Mapping type names can't start with underscores (_)

Fluent Bit v1.5 changed the default mapping type from flb_type to _doc, matching the recommendation from Elasticsearch for version 6.2 and greater (see commit with rationale).

This doesn't work in Elasticsearch versions 5.6 through 6.1 (discussion and fix).

Ensure you set an explicit map such as doc or flb_type in the configuration, as seen on the last line:

[OUTPUT]
    Name  es
    Match *
    Host  vpc-test-domain-ke7thhzoo7jawsrhmm6mb7ite7y.us-west-2.es.amazonaws.com
    Port  443
    Index my_index
    AWS_Auth On
    AWS_Region us-west-2
    tls   On
    Type  doc

Validation failures

In Fluent Bit v1.8.2 and greater, Fluent Bit started using create method (instead of index) for data submission. This makes Fluent Bit compatible with Datastream, introduced in Elasticsearch 7.9. You might see errors like:

Validation Failed: 1: an id must be provided if version type or value are set

If you see action_request_validation_exception errors on your pipeline with Fluent Bit versions greater than v1.8.2, correct them by turning on Generate_ID as follows:

[OUTPUT]
    Name es
    Match *
    Host  192.168.12.1
    Generate_ID on

Logstash_Prefix_Key

The following snippet demonstrates using the namespace name as extracted by the kubernetes filter as logstash prefix:

[OUTPUT]
    Name es
    Match *
    # ...
    Logstash_Prefix logstash
    Logstash_Prefix_Key $kubernetes["namespace_name"]
    # ...

For records that don't have the field kubernetes.namespace_name, the default prefix logstash will be used.

Stream logs to Dynatrace by utilizing the http plugin to send data to the Dynatrace generic log ingest API.

Configuration parameters

Getting started

To get started with sending logs to Dynatrace:

1. Get a Dynatrace API token with the logs.ingest (Ingest Logs) scope.

2. Determine your Dynatrace environment ID.

3. In your main Fluent Bit configuration file, append the following Output section:

   Copy

   [OUTPUT]
       name         http
       match        *
       header       Content-Type application/json; charset=utf-8
       header       Authorization Api-Token {your-API-token-here}
       allow_duplicated_headers false
       host         {your-environment-id}.live.dynatrace.com
       Port         443
       URI          /api/v2/logs/ingest
       Format       json
       json_date_format iso8601
       json_date_key timestamp
       tls          On
       tls.verify   On

References

• Dynatrace Fluent Bit documentation

• Fluent Bit integration in Dynatrace Hub

• Video: Stream a Log File to Dynatrace using Fluent Bit

• Blog: Easily stream logs from Fluent Bit to Dynatrace

Azure Logs Ingestion plugin allows you ingest your records using Logs Ingestion API in Azure Monitor to supported Azure tables or to custom tables that you create.

The Logs ingestion API requires the following components:

• A Data Collection Endpoint (DCE)

• A Data Collection Rule (DCR) and

• A Log Analytics Workspace

Note: According to this document, all resources should be in the same region.

To get more details about how to setup these components, please refer to the following documentations:

• Azure Logs Ingestion API

• Send data to Azure Monitor Logs with Logs ingestion API (setup DCE, DCR and Log Analytics)

Configuration Parameters

Getting Started

To send records into an Azure Log Analytics using Logs Ingestion API the following resources needs to be created:

• A Data Collection Endpoint (DCE) for ingestion

• A Data Collection Rule (DCR) for data transformation

• Either an Azure tables or custom tables

• An app registration with client secrets (for DCR access).

You can follow this guideline to setup the DCE, DCR, app registration and a custom table.

Configuration File

Use this configuration to quickly get started:

[INPUT]
    Name    tail
    Path    /path/to/your/sample.log
    Tag     sample
    Key     RawData
# Or use other plugins Plugin
# [INPUT]
#     Name    cpu
#     Tag     sample

[FILTER]
    Name modify
    Match sample
    # Add a json key named "Application":"fb_log"
    Add Application fb_log

# Enable this section to see your json-log format
#[OUTPUT]
#    Name stdout
#    Match *
[OUTPUT]
    Name            azure_logs_ingestion
    Match           sample
    client_id       XXXXXXXX-xxxx-yyyy-zzzz-xxxxyyyyzzzzxyzz
    client_secret   some.secret.xxxzzz
    tenant_id       XXXXXXXX-xxxx-yyyy-zzzz-xxxxyyyyzzzzxyzz
    dce_url         https://log-analytics-dce-XXXX.region-code.ingest.monitor.azure.com
    dcr_id          dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    table_name      ladcr_CL
    time_generated  true
    time_key        Time
    Compress        true

Setup your DCR transformation accordingly based on the json output from fluent-bit's pipeline (input, parser, filter, output).

The file output plugin allows to write the data received through the input plugin to file.

Configuration Parameters

The plugin supports the following configuration parameters:

Format

out_file format

Output time, tag and json records. There is no configuration parameters for out_file.

tag: [time, {"key1":"value1", "key2":"value2", "key3":"value3"}]

plain format

Output the records as JSON (without additional tag and timestamp attributes). There is no configuration parameters for plain format.

{"key1":"value1", "key2":"value2", "key3":"value3"}

csv format

Output the records as csv. Csv supports an additional configuration parameter.

time[delimiter]"value1"[delimiter]"value2"[delimiter]"value3"

ltsv format

Output the records as LTSV. LTSV supports an additional configuration parameter.

field1[label_delimiter]value1[delimiter]field2[label_delimiter]value2\n

template format

Output the records using a custom format template.

This accepts a formatting template and fills placeholders using corresponding values in a record.

For example, if you set up the configuration as below:

[INPUT]
  Name mem

[OUTPUT]
  Name file
  Format template
  Template {time} used={Mem.used} free={Mem.free} total={Mem.total}

You will get the following output:

1564462620.000254 used=1045448 free=31760160 total=32805608

Getting Started

You can run the plugin from the command line or through the configuration file:

Command Line

From the command line you can let Fluent Bit count up a data with the following options:

$ fluent-bit -i cpu -o file -p path=output.txt

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name cpu
    Tag  cpu

[OUTPUT]
    Name file
    Match *
    Path output_dir

The Amazon S3 output plugin lets you ingest records into the S3 cloud object store.

The plugin can upload data to S3 using the multipart upload API or PutObject. Multipart is the default and is recommended. Fluent Bit will stream data in a series of parts. This limits the amount of data buffered on disk at any point in time. By default, every time 5 MiB of data have been received, a new part will be uploaded. The plugin can create files up to gigabytes in size from many small chunks or parts using the multipart API. All aspects of the upload process are configurable.

The plugin lets you specify a maximum file size, and a timeout for uploads. A file will be created in S3 when the maximum size or the timeout is reached, whichever comes first.

Records are stored in files in S3 as newline delimited JSON.

See AWS Credentials for details about fetching AWS credentials.

Configuration Parameters

TLS / SSL

To skip TLS verification, set tls.verify as false. For more details about the properties available and general configuration, refer to TLS/SSL.

Permissions

The plugin requires the following AWS IAM permissions:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "s3:PutObject"
        ],
        "Resource": "*"
    }]
}

Differences between S3 and other Fluent Bit outputs

The S3 output plugin is used to upload large files to an Amazon S3 bucket, while most other outputs which send many requests to upload data in batches of a few megabytes or less.

When Fluent Bit receives logs, it stores them in chunks, either in memory or the filesystem depending on your settings. Chunks are usually around 2 MB in size. Fluent Bit sends chunks, in order, to each output that matches their tag. Most outputs then send the chunk immediately to their destination. A chunk is sent to the output's flush callback function, which must return one of FLB_OK, FLB_RETRY, or FLB_ERROR. Fluent Bit keeps count of the return values from each output's flush callback function. These counters are the data source for Fluent Bit's error, retry, and success metrics available in Prometheus format through its monitoring interface.

The S3 output plugin conforms to the Fluent Bit output plugin specification. Since S3's use case is to upload large files (over 2 MB), its behavior is different. S3's flush callback function buffers the incoming chunk to the filesystem, and returns an FLB_OK. This means Prometheus metrics available from the Fluent Bit HTTP server are meaningless for S3. In addition, the storage.total_limit_size parameter is not meaningful for S3 since it has its own buffering system in the store_dir. Instead, use store_dir_limit_size. S3 requires a writeable filesystem. Running Fluent Bit on a read-only filesystem won't work with the S3 output.

S3 uploads primarily initiate using the S3 timer callback function, which runs separately from its flush.

S3 has its own buffering system and its own callback to upload data, so the normal sequential data ordering of chunks provided by the Fluent Bit engine may be compromised. S3 has the presevere_data_ordering option which ensures data is uploaded in the original order it was collected by Fluent Bit.

Summary: Uniqueness in S3 Plugin

• The HTTP Monitoring interface output metrics are not meaningful for S3. AWS understands that this is non-ideal; we have opened an issue with a design to allow S3 to manage its own output metrics.

• You must use store_dir_limit_size to limit the space on disk used by S3 buffer files.

• The original ordering of data inputted to Fluent Bit may not be preserved unless you enable preserve_data_ordering On.

S3 Key Format and Tag Delimiters

In Fluent Bit, all logs have an associated tag. The s3_key_format option lets you inject the tag into the S3 key using the following syntax:

• $TAG: The full tag.

• $TAG[n]: The nth part of the tag (index starting at zero). This syntax is copied from the rewrite tag filter. By default, “parts” of the tag are separated with dots, but you can change this with s3_key_format_tag_delimiters.

In the following example, assume the date is January 1st, 2020 00:00:00 and the tag associated with the logs in question is my_app_name-logs.prod.

[OUTPUT]
    Name  s3
    Match *
    bucket                       my-bucket
    region                       us-west-2
    total_file_size              250M
    s3_key_format                /$TAG[2]/$TAG[0]/%Y/%m/%d/%H/%M/%S/$UUID.gz
    s3_key_format_tag_delimiters .-

With the delimiters as . and -, the tag splits into parts as follows:

• $TAG[0] = my_app_name

• $TAG[1] = logs

• $TAG[2] = prod

The key in S3 will be /prod/my_app_name/2020/01/01/00/00/00/bgdHN1NM.gz.

Allowing a file extension in the S3 Key Format with $UUID

The Fluent Bit S3 output was designed to ensure that previous uploads will never be overwritten by a subsequent upload. The s3_key_format supports time formatters, $UUID, and $INDEX. $INDEX is special because it is saved in the store_dir. If you restart Fluent Bit with the same disk, it can continue incrementing the index from its last value in the previous run.

For files uploaded with the PutObject API, the S3 output requires that a unique random string be present in the S3 key. Many of the use cases for PutObject uploads involve a short time period between uploads, so a timestamp in the S3 key may not be unique enough between uploads. For example, if you only specify minute granularity timestamps in the S3 key, with a small upload size, it is possible to have two uploads that have timestamps set in the same minute. This requirement can be disabled with static_file_path On.

The PutObject API is used in these cases:

• When you explicitly set use_put_object On.

• On startup when the S3 output finds old buffer files in the store_dir from a previous run and attempts to send all of them at once.

• On shutdown. To prevent data loss the S3 output attempts to send all currently buffered data at once.

You should always specify $UUID somewhere in your S3 key format. Otherwise, if the PutObject API is used, S3 appends a random eight-character UUID to the end of your S3 key. This means that a file extension set at the end of an S3 key will have the random UUID appended to it. Disabled this with static_file_path On.

For example, we attempt to set a .gz extension without specifying $UUID:

[OUTPUT]
    Name  s3
    Match *
    bucket                       my-bucket
    region                       us-west-2
    total_file_size              50M
    use_put_object               Off
    compression                  gzip
    s3_key_format                /$TAG/%Y/%m/%d/%H_%M_%S.gz

In the case where pending data is uploaded on shutdown, if the tag was app, the S3 key in the S3 bucket might be:

/app/2022/12/25/00_00_00.gz-apwgylqg

The S3 output appended a random string to the file extension, since this upload on shutdown used the PutObject API.

There are two ways of disabling this behavior:

• Use static_file_path:

  Copy

  [OUTPUT]
      Name  s3
      Match *
      bucket                       my-bucket
      region                       us-west-2
      total_file_size              50M
      use_put_object               Off
      compression                  gzip
      s3_key_format                /$TAG/%Y/%m/%d/%H_%M_%S.gz
      static_file_path             On

• Explicitly define where the random UUID will go in the S3 key format:

  Copy

  [OUTPUT]
      Name  s3
      Match *
      bucket                       my-bucket
      region                       us-west-2
      total_file_size              50M
      use_put_object               Off
      compression                  gzip
      s3_key_format                /$TAG/%Y/%m/%d/%H_%M_%S/$UUID.gz

Reliability

The store_dir is used to temporarily store data before upload. If Fluent Bit stops suddenly, it will try to send all data and complete all uploads before it shuts down. If it can not send some data, on restart it will look in the store_dir for existing data and try to send it.

Multipart uploads are ideal for most use cases because they allow the plugin to upload data in small chunks over time. For example, 1 GB file can be created from 200 5 MB chunks. While the file size in S3 will be 1 GB, only 5 MB will be buffered on disk at any one point in time.

One drawback to multipart uploads is that the file and data aren't visible in S3 until the upload is completed with a CompleteMultipartUpload call. The plugin attempts to make this call whenever Fluent Bit is shut down to ensure your data is available in S3. It also stores metadata about each upload in the store_dir, ensuring that uploads can be completed when Fluent Bit restarts (assuming it has access to persistent disk and the store_dir files will still be present on restart).

Using S3 without persisted disk

If you run Fluent Bit in an environment without persistent disk, or without the ability to restart Fluent Bit and give it access to the data stored in the store_dir from previous executions, some considerations apply. This might occur if you run Fluent Bit on AWS Fargate.

In these situations, we recommend using the PutObject API and sending data frequently, to avoid local buffering as much as possible. This will limit data loss in the event Fluent Bit is killed unexpectedly.

The following settings are recommended for this use case:

[OUTPUT]
    Name s3
    Match *
    bucket your-bucket
    region us-east-1
    total_file_size 1M
    upload_timeout 1m
    use_put_object On

S3 Multipart Uploads

With use_put_object Off (default), S3 will attempt to send files using multipart uploads. For each file, S3 first calls CreateMultipartUpload, then a series of calls to UploadPart for each fragment (targeted to be upload_chunk_size bytes), and finally CompleteMultipartUpload to create the final file in S3.

Fallback to PutObject

S3 requires each UploadPart fragment to be at least 5,242,880 bytes, otherwise the upload is rejected.

The S3 output must sometimes fallback to the PutObject API.

Uploads are triggered by these settings:

• total_file_size and upload_chunk_size: When S3 has buffered data in the store_dir that meets the desired total_file_size (for use_put_object On) or the upload_chunk_size (for Multipart), it will trigger an upload operation.

• upload_timeout: Whenever locally buffered data has been present on the filesystem in the store_dir longer than the configured upload_timeout, it will be sent even when the desired byte size hasn't been reached. If you configure a small upload_timeout, your files may be smaller than the total_file_size. The timeout is evaluated against the time at which S3 started buffering data for each unqiue tag (that is, the time when new data was buffered for the unique tag after the last upload). The timeout is also evaluated against the CreateMultipartUpload time, so a multipart upload will be completed after upload_timeout has elapsed, even if the desired size has not yet been reached.

If your upload_timeout triggers an upload before the pending buffered data reaches the upload_chunk_size, it may be too small for a multipart upload. S3 will fallback to use the PutObject API.

When you enable compression, S3 applies the compression algorithm at send time. The size settings trigger uploads based on the size of buffered data, not the final compressed size. It's possible that after compression, buffered data no longer meets the required minimum S3 UploadPart size. If this occurs, you will see a log message like:

[ info] [output:s3:s3.0] Pre-compression upload_chunk_size= 5630650, After
compression, chunk is only 1063320 bytes, the chunk was too small, using PutObject to upload

If you encounter this frequently, use the numbers in the messages to guess your compression factor. In this example, the buffered data was reduced from 5,630,650 bytes to 1,063,320 bytes. The compressed size is one-fifth the actual data size. Configuring upload_chunk_size 30M should ensure each part is large enough after compression to be over the minimum required part size of 5,242,880 bytes.

The S3 API allows the last part in an upload to be less than the 5,242,880 byte minimum. If a part is too small for an existing upload, the S3 output will upload that part and then complete the upload.

upload_timeout constrains total multipart upload time for a single file

The upload_timeout evaluated against the CreateMultipartUpload time. A multipart upload will be completed after upload_timeout elapses, even if the desired size has not yet been reached.

Completing uploads

When CreateMultipartUpload is called, an UploadID is returned. S3 stores these IDs for active uploads in the store_dir. Until CompleteMultipartUpload is called, the uploaded data isn't visible in S3.

On shutdown, S3 output attempts to complete all pending uploads. If an upload fails to complete, the ID remains buffered in the store_dir in a directory called multipart_upload_metadata. If you restart the S3 output with the same store_dir it will discover the old UploadIDs and complete the pending uploads. The S3 documentation has suggestions on discovering and deleting or completing dangling uploads in your buckets.

Usage with MinIO

MinIO is a high-performance, S3 compatible object storage and you can build your app with S3 functionality without S3.

The following example runs a MinIO server at localhost:9000, and create a bucket of your-bucket.

Example:

[OUTPUT]
   Name s3
   Match *
   bucket your-bucket
   endpoint http://localhost:9000

The records store in the MinIO server.

Get Started

To send records into Amazon S3, you can run the plugin from the command line or through the configuration file.

Command Line

The S3 plugin reads parameters from the command line through the -p argument:

fluent-bit -i cpu -o s3 -p bucket=my-bucket -p region=us-west-2 -p -m '*' -f 1

Configuration File

In your main configuration file append the following Output section:

[OUTPUT]
    Name s3
    Match *
    bucket your-bucket
    region us-east-1
    store_dir /home/ec2-user/buffer
    total_file_size 50M
    upload_timeout 10m

An example using PutObject instead of multipart:

[OUTPUT]
    Name s3
    Match *
    bucket your-bucket
    region us-east-1
    store_dir /home/ec2-user/buffer
    use_put_object On
    total_file_size 10M
    upload_timeout 10m

AWS for Fluent Bit

Amazon distributes a container image with Fluent Bit and plugins.

GitHub

github.com/aws/aws-for-fluent-bit

Amazon ECR Public Gallery

Our images are available in the Amazon ECR Public Gallery as aws-for-fluent-bit.

You can download images with different tags using the following command:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:<tag>

For example, you can pull the image with latest version with:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

If you see errors for image pull limits, try signing in to public ECR with your AWS credentials:

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

See the Amazon ECR Public official documentation for more details.

Docker Hub

amazon/aws-for-fluent-bit is also available from the Docker Hub.

Amazon ECR

Use our SSM Public Parameters to find the Amazon ECR image URI in your region:

aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/

For more information, see the AWS for Fluent Bit GitHub repo.

Advanced usage

Use Apache Arrow for in-memory data processing

With Fluent Bit v1.8 or greater, the Amazon S3 plugin includes the support for Apache Arrow. Support isn't enabled by default, and has a dependency on a shared version of libarrow.

To use this feature, FLB_ARROW must be turned on at compile time. Use the following commands:

cd build/
cmake -DFLB_ARROW=On ..
cmake --build .

After being compiled, Fluent Bit can upload incoming data to S3 in Apache Arrow format.

For example:

[INPUT]
    Name cpu

[OUTPUT]
    Name s3
    Bucket your-bucket-name
    total_file_size 1M
    use_put_object On
    upload_timeout 60s
    Compression arrow

Setting Compression to arrow makes Fluent Bit convert payload into Apache Arrow format.

Load, analyze, and process stored data using popular data processing tools such as Python pandas, Apache Spark and Tensorflow.

The following example uses pyarrow to analyze the uploaded data:

>>> import pyarrow.feather as feather
>>> import pyarrow.fs as fs
>>>
>>> s3 = fs.S3FileSystem()
>>> file = s3.open_input_file("my-bucket/fluent-bit-logs/cpu.0/2021/04/27/09/36/15-object969o67ZF")
>>> df = feather.read_feather(file)
>>> print(df.head())
   date  cpu_p  user_p  system_p  cpu0.p_cpu  cpu0.p_user  cpu0.p_system
0  2021-04-27T09:33:53.539346Z    1.0     1.0       0.0         1.0          1.0            0.0
1  2021-04-27T09:33:54.539330Z    0.0     0.0       0.0         0.0          0.0            0.0
2  2021-04-27T09:33:55.539305Z    1.0     0.0       1.0         1.0          0.0            1.0
3  2021-04-27T09:33:56.539430Z    0.0     0.0       0.0         0.0          0.0            0.0
4  2021-04-27T09:33:57.539803Z    0.0     0.0       0.0         0.0          0.0            0.0

The influxdb output plugin, allows to flush your records into a InfluxDB time series database. The following instructions assumes that you have a fully operational InfluxDB service running in your system.

Configuration Parameters

TLS / SSL

InfluxDB output plugin supports TLS/SSL, for more details about the properties available and general configuration, please refer to the TLS/SSL section.

Getting Started

In order to start inserting records into an InfluxDB service, you can run the plugin from the command line or through the configuration file:

Command Line

The influxdb plugin, can read the parameters from the command line in two ways, through the -p argument (property) or setting them directly through the service URI. The URI format is the following:

influxdb://host:port

Using the format specified, you could start Fluent Bit through:

$ fluent-bit -i cpu -t cpu -o influxdb://127.0.0.1:8086 -m '*'

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name  cpu
    Tag   cpu

[OUTPUT]
    Name          influxdb
    Match         *
    Host          127.0.0.1
    Port          8086
    Database      fluentbit
    Sequence_Tag  _seq

Tagging

Basic example of Tag_Keys usage:

[INPUT]
    Name            tail
    Tag             apache.access
    parser          apache2
    path            /var/log/apache2/access.log

[OUTPUT]
    Name          influxdb
    Match         *
    Host          127.0.0.1
    Port          8086
    Database      fluentbit
    Sequence_Tag  _seq
    # make tags from method and path fields
    Tag_Keys      method path

With Auto_Tags=On in this example cause error, because every parsed field value type is string. Best usage of this option in metrics like record where one or more field value is not string typed.

Basic example of Tags_List_Key usage:

[INPUT]
    Name              dummy
    # tagged fields: level, ID, businessObjectID, status
    Dummy             {"msg": "Transfer completed", "level": "info", "ID": "1234", "businessObjectID": "qwerty", "status": "OK", "tags": ["ID", "businessObjectID"]}

[OUTPUT]
    Name          influxdb
    Match         *
    Host          127.0.0.1
    Port          8086
    Bucket        My_Bucket
    Org           My_Org
    Sequence_Tag  _seq
    HTTP_Token    My_Token
    # tag all fields inside tags string array
    Tags_List_Enabled True
    Tags_List_Key tags
    # tag level, status fields
    Tag_Keys level status

Testing

Before to start Fluent Bit, make sure the target database exists on InfluxDB, using the above example, we will insert the data into a fluentbit database.

1. Create database

Log into InfluxDB console:

$ influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 1.1.0
InfluxDB shell version: 1.1.0
>

Create the database:

> create database fluentbit
>

Check the database exists:

> show databases
name: databases
name
----
_internal
fluentbit

>

2. Run Fluent Bit

The following command will gather CPU metrics from the system and send the data to InfluxDB database every five seconds:

$ bin/fluent-bit -i cpu -t cpu -o influxdb -m '*'

Note that all records coming from the cpu input plugin, have a tag cpu, this tag is used to generate the measurement in InfluxDB

3. Query the data

From InfluxDB console, choose your database:

> use fluentbit
Using database fluentbit

Now query some specific fields:

> SELECT cpu_p, system_p, user_p FROM cpu
name: cpu
time                  cpu_p   system_p    user_p
----                  -----   --------    ------
1481132860000000000   2.75        0.5      2.25
1481132861000000000   2           0.5      1.5
1481132862000000000   4.75        1.5      3.25
1481132863000000000   6.75        1.25     5.5
1481132864000000000   11.25       3.75     7.5

The CPU input plugin gather more metrics per CPU core, in the above example we just selected three specific metrics. The following query will give a full result:

> SELECT * FROM cpu

4. View tags

Query tagged keys:

> SHOW TAG KEYS ON fluentbit FROM "apache.access"
name: apache.access
tagKey
------
_seq
method
path

And now query method key values:

> SHOW TAG VALUES ON fluentbit FROM "apache.access" WITH KEY = "method"
name: apache.access
key    value
---    -----
method "MATCH"
method "POST"

The kafka-rest output plugin, allows to flush your records into a server. The following instructions assumes that you have a fully operational Kafka REST Proxy and Kafka services running in your environment.

Configuration Parameters

TLS / SSL

Kafka REST Proxy output plugin supports TLS/SSL, for more details about the properties available and general configuration, please refer to the section.

Getting Started

In order to insert records into a Kafka REST Proxy service, you can run the plugin from the command line or through the configuration file:

Command Line

The kafka-rest plugin, can read the parameters from the command line in two ways, through the -p argument (property), e.g:

Configuration File

In your main configuration file append the following Input & Output sections:

The http output plugin allows to flush your records into a HTTP endpoint. For now the functionality is pretty basic and it issues a POST request with the data records in (or JSON) format.

Configuration Parameters

TLS / SSL

Getting Started

In order to insert records into a HTTP server, you can run the plugin from the command line or through the configuration file:

Command Line

The http plugin, can read the parameters from the command line in two ways, through the -p argument (property) or setting them directly through the service URI. The URI format is the following:

Using the format specified, you could start Fluent Bit through:

Configuration File

In your main configuration file, append the following Input & Output sections:

By default, the URI becomes tag of the message, the original tag is ignored. To retain the tag, multiple configuration sections have to be made based and flush to different URIs.

Another approach we also support is the sending the original message tag in a configurable header. It's up to the receiver to do what it wants with that header field: parse it and use it as the tag for example.

To configure this behaviour, add this config:

Provided you are using Fluentd as data receiver, you can combine in_http and out_rewrite_tag_filter to make use of this HTTP header.

Notice how we override the tag, which is from URI path, with our custom header

Example : Add a header

Example : Sumo Logic HTTP Collector

Suggested configuration for Sumo Logic using json_lines with iso8601 timestamps. The PrivateKey is specific to a configured HTTP collector.

GELF is Extended Log Format. The GELF output plugin allows to send logs in GELF format directly to a Graylog input using TLS, TCP or UDP protocols.

The following instructions assumes that you have a fully operational Graylog server running in your environment.

Configuration Parameters

According to , there are some mandatory and optional fields which are used by Graylog in GELF format. These fields are determined with Gelf\*_Key_ key in this plugin.

TLS / SSL

Notes

• If you're using Fluent Bit to collect Docker logs, note that Docker places your log in JSON under key log. So you can set log as your Gelf_Short_Message_Key to send everything in Docker logs to Graylog. In this case, you need your log value to be a string; so don't parse it using JSON parser.

• The order of looking up the timestamp in this plugin is as follows:

  1. Value of Gelf_Timestamp_Key provided in configuration

  2. Value of timestamp key

  4. Timestamp does not set by Fluent Bit. In this case, your Graylog server will set it to the current timestamp (now).

• The version of GELF message is also mandatory and Fluent Bit sets it to 1.1 which is the current latest version of GELF.

• If you use udp as transport protocol and set Compress to true, Fluent Bit compresses your packets in GZIP format, which is the default compression that Graylog offers. This can be used to trade more CPU load for saving network bandwidth.

Configuration File Example

If you're using Fluent Bit for shipping Kubernetes logs, you can use something like this as your configuration file:

By default, GELF tcp uses port 12201 and Docker places your logs in /var/log/containers directory. The logs are placed in value of the log key. For example, this is a log saved by Docker:

Now, this is what happens to this log:

1. Fluent Bit GELF plugin adds "version": "1.1" to it.

3. We used this data key as Gelf_Short_Message_Key; so GELF plugin changes it to short_message.

5. Timestamp is generated.

Finally, this is what our Graylog server input sees:

is an intuitive cloud based log management system that provides you an easy interface to query your logs once they are stored.

The Fluent Bit logdna output plugin allows you to send your log or events to a compliant service like:

Before to get started with the plugin configuration, make sure to obtain the proper account to get access to the service. You can start with a free trial in the following link:

Configuration Parameters

Auto Enrichment & Data Discovery

One of the features of Fluent Bit + LogDNA integration is the ability to auto enrich each record with further context.

When the plugin process each record (or log), it tries to lookup for specific key names that might contain specific context for the record in question, the following table describe the keys and the discovery logic:

Getting Started

The following configuration example, will emit a dummy example record and ingest it on LogDNA. Copy and paste the following content in a file called logdna.conf:

run Fluent Bit with the new configuration file:

Fluent Bit output:

Your record will be available and visible in your LogDNA dashboard after a few seconds.

Query your Data in LogDNA

In your LogDNA dashboard, go to the top filters and mark the Tags aa and bb, then you will be able to see your records as the example below:

The Chronicle output plugin allows ingesting security logs into service. This connector is designed to send unstructured security logs.

Google Cloud Configuration

Fluent Bit streams data into an existing Google Chronicle tenant using a service account that you specify. Therefore, before using the Chronicle output plugin, you must create a service account, create a Google Chronicle tenant, authorize the service account to write to the tenant, and provide the service account credentials to Fluent Bit.

Creating a Service Account

To stream security logs into Google Chronicle, the first step is to create a Google Cloud service account for Fluent Bit:

Creating a Tenant of Google Chronicle

Fluent Bit does not create a tenant of Google Chronicle for your security logs, so you must create this ahead of time.

Retrieving Service Account Credentials

Fluent Bit's Chronicle output plugin uses a JSON credentials file for authentication credentials. Download the credentials file by following these instructions:

Configurations Parameters

Configuration File

If you are using a Google Cloud Credentials File, the following configuration is enough to get you started:

is multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate.

The Fluent Bit loki built-in output plugin allows you to send your log or events to a Loki service. It supports data enrichment with Kubernetes labels, custom label keys and Tenant ID within others.

Be aware there is a separate Golang output plugin provided by with different configuration options.

Configuration Parameters

Labels

Loki store the record logs inside Streams, a stream is defined by a set of labels, at least one label is required.

Fluent Bit implements a flexible mechanism to set labels by using fixed key/value pairs of text but also allowing to set as labels certain keys that exists as part of the records that are being processed. Consider the following JSON record (pretty printed for readability):

If you decide that your Loki Stream will be composed by two labels called job and the value of the record key called stream , your labels configuration properties might look as follows:

When processing above's configuration, internally the ending labels for the stream in question becomes:

Another feature of Labels management is the ability to provide custom key names, using the same record accessor pattern we can specify the key name manually and let the value to be populated automatically at runtime, e.g:

When processing that new configuration, the internal labels will be:

Using the label_keys property

The additional configuration property called label_keys allow to specify multiple record keys that needs to be placed as part of the outgoing Stream Labels, yes, this is a similar feature than the one explained above in the labels property. Consider this as another way to set a record key in the Stream, but with the limitation that you cannot use a custom name for the key value.

The following configuration examples generate the same Stream Labels:

the above configuration accomplish the same than this one:

both will generate the following Streams label:

Using the label_map_path property

The configuration property label_map_path is to read a JSON file that defines how to extract labels from each record.

The file should contain a JSON object. Each keys define how to get label value from a nested record. Each values are used as label names.

The following configuration examples generate the same Stream Labels:

map.json:

The following configuration examples generate the same Stream Labels:

the above configuration accomplish the same than this one:

both will generate the following Streams label:

Kubernetes & Labels

Note that if you are running in a Kubernetes environment, you might want to enable the option auto_kubernetes_labels which will auto-populate the streams with the Pod labels for you. Consider the following configuration:

Based in the JSON example provided above, the internal stream labels will be:

Drop Single Key

If there is only one key remaining after removing keys, you can use the drop_single_key property to send its value to Loki, rather than a single key=value pair.

Consider this simple JSON example:

If the value is a string, line_format is json, and drop_single_key is true, it will be sent as a quoted string.

The outputted line would show in Loki as:

If drop_single_key is raw, or line_format is key_value, it will show in Loki as:

If you want both structured JSON and plain-text logs in Loki, you should set drop_single_key to raw and line_format to json. Loki does not interpret a quoted string as valid JSON, and so to remove the quotes without drop_single_key set to raw, you would need to use a query like this:

If drop_single_key is off, it will show in Loki as:

You can get the same behavior this flag provides in Loki with drop_single_key set to off with this query:

Structured metadata

The following configuration:

• Defines fixed values for the cluster and region labels.

• Uses the record accessor pattern to set the namespace label to the namespace name as determined by the Kubernetes metadata filter (not shown).

• Uses a structured metadata field to hold the Kubernetes pod name.

Other common uses for structured metadata include trace and span IDs, process and thread IDs, and log levels.

Structured metadata is officially supported starting with Loki 3.0, and shouldn't be used with Loki deployments prior to Loki 3.0.

Networking and TLS Configuration

This plugin inherit core Fluent Bit features to customize the network behavior and optionally enable TLS in the communication channel. For more details about the specific options available refer to the following articles:

Note that all options mentioned in the articles above must be enabled in the plugin configuration in question.

Fluent Bit + Grafana Cloud

An example configuration - make sure to set the credentials and ensure the host URL matches the correct one for your deployment:

Getting Started

The following configuration example, will emit a dummy example record and ingest it on Loki . Copy and paste the following content into a file called out_loki.conf:

run Fluent Bit with the new configuration file:

Fluent Bit output:

Forward is the protocol used by to route messages between peers. The forward output plugin provides interoperability between and . There are no configuration steps required besides specifying where is located, which can be a local or a remote destination.

This plugin offers two different transports and modes:

• Forward (TCP): It uses a plain TCP connection.

• Secure Forward (TLS): when TLS is enabled, the plugin switch to Secure Forward mode.

Configuration Parameters

The following parameters are mandatory for either Forward for Secure Forward modes:

Secure Forward Mode Configuration Parameters

Forward Setup

That configuration file specifies that it will listen for TCP connections on the port 24224 through the forward input type. Then for every message with a fluent_bit TAG, will print the message to the standard output.

Fluent Bit + Forward Setup

Fluent Bit + Secure Forward Setup

DISCLAIMER: the following example does not consider the generation of certificates for best practice on production environments.

Fluent Bit

Paste this content in a file called flb.conf:

Fluentd

Paste this content in a file called fld.conf:

If you're using Fluentd v1, set up it as below:

Test Communication

Start Fluentd:

Start Fluent Bit:

After five seconds, Fluent Bit will write records to Fluentd. In Fluentd output you will see a message like this:

Kafka output plugin allows to ingest your records into an service. This plugin use the official (built-in dependency)

Configuration Parameters

Setting rdkafka.log.connection.close to false and rdkafka.request.required.acks to 1 are examples of recommended settings of librdfkafka properties.

Getting Started

In order to insert records into Apache Kafka, you can run the plugin from the command line or through the configuration file:

Command Line

The kafka plugin can read parameters through the -p argument (property), e.g:

Configuration File

In your main configuration file append the following Input & Output sections:

Avro Support

Fluent-bit comes with support for avro encoding for the out_kafka plugin. Avro support is optional and must be activated at build-time by using a build def with cmake: -DFLB_AVRO_ENCODER=On such as in the following example which activates:

• out_kafka with avro encoding

• fluent-bit's prometheus

• metrics via an embedded http endpoint

• debugging support

• builds the test suites

Kafka Configuration File with Avro Encoding

This is example fluent-bit config tails kubernetes logs, decorates the log lines with kubernetes metadata via the kubernetes filter, and then sends the fully decorated log lines to a kafka broker encoded with a specific avro schema.

Kafka Configuration File with Raw format

This example Fluent Bit configuration file creates example records with the payloadkey and msgkey keys. The msgkey value is used as the Kafka message key, and the payloadkey value as the payload.