1 of 45

Outputs

Amazon CloudWatch

Send logs and metrics to Amazon CloudWatch

The Amazon CloudWatch output plugin allows to ingest your records into the CloudWatch Logs service. Support for CloudWatch Metrics is also provided via EMF.

This is the documentation for the core Fluent Bit CloudWatch plugin written in C. It can replace the aws/amazon-cloudwatch-logs-for-fluent-bit Golang Fluent Bit plugin released last year. The Golang plugin was named cloudwatch; this new high performance CloudWatch plugin is called cloudwatch_logs to prevent conflicts/confusion. Check the amazon repo for the Golang plugin for details on the deprecation/migration plan for the original plugin.

See here for details on how AWS credentials are fetched.

Configuration Parameters

Getting Started

In order to send records into Amazon Cloudwatch, you can run the plugin from the command line or through the configuration file:

Command Line

The cloudwatch plugin, can read the parameters from the command line through the -p argument (property), e.g:

$ fluent-bit -i cpu -o cloudwatch_logs -p log_group_name=group -p log_stream_name=stream -p region=us-west-2 -m '*' -f 1

Configuration File

In your main configuration file append the following Output section:

[OUTPUT]
    Name cloudwatch_logs
    Match   *
    region us-east-1
    log_group_name fluent-bit-cloudwatch
    log_stream_prefix from-fluent-bit-
    auto_create_group On

Intergration with Localstack (Cloudwatch Logs)

For an instance of Localstack running at http://localhost:4566, the following configuration needs to be added to the [OUTPUT] section:

endpoint localhost
port 4566

Any testing credentials can be exported as local variables, such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Permissions

The following AWS IAM permissions are required to use this plugin:

{
	"Version": "2012-10-17",
	"Statement": [{
		"Effect": "Allow",
		"Action": [
			"logs:CreateLogStream",
			"logs:CreateLogGroup",
			"logs:PutLogEvents"
		],
		"Resource": "*"
	}]
}

Log Stream and Group Name templating using record_accessor syntax

Sometimes, you may want the log group or stream name to be based on the contents of the log record itself. This plugin supports templating log group and stream names using Fluent Bit record_accessor syntax.

Here is an example usage, for a common use case- templating log group and stream names based on Kubernetes metadata.

Recall that the kubernetes filter can add metadata which will look like the following:

kubernetes: {
    annotations: {
        "kubernetes.io/psp": "eks.privileged"
    },
    container_hash: "<some hash>",
    container_name: "myapp",
    docker_id: "<some id>",
    host: "ip-10-1-128-166.us-east-2.compute.internal",
    labels: {
        app: "myapp",
        "pod-template-hash": "<some hash>"
    },
    namespace_name: "my-namespace",
    pod_id: "198f7dd2-2270-11ea-be47-0a5d932f5920",
    pod_name: "myapp-5468c5d4d7-n2swr"
}

Using record_accessor, we can build a template based on this object.

Here is our output configuration:

[OUTPUT]
    Name cloudwatch_logs
    Match   *
    region us-east-1
    log_group_name fallback-group
    log_stream_prefix fallback-stream
    auto_create_group On
    log_group_template application-logs-$kubernetes['host'].$kubernetes['namespace_name']
    log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']

With the above kubernetes metadata, the log group name will be application-logs-ip-10-1-128-166.us-east-2.compute.internal.my-namespace. And the log stream name will be myapp-5468c5d4d7-n2swr.myapp.

If the kubernetes structure is not found in the log record, then the log_group_name and log_stream_prefix will be used instead, and Fluent Bit will log an error like:

[2022/06/30 06:09:29] [ warn] [record accessor] translation failed, root key=kubernetes

Limitations of record_accessor syntax

Notice in the example above, that the template values are separated by dot characters. This is important; the Fluent Bit record_accessor library has a limitation in the characters that can separate template variables- only dots and commas (. and ,) can come after a template variable. This is because the templating library must parse the template and determine the end of a variable.

Assume that your log records contain the metadata keys container_name and task. The following would be invalid templates because the two template variables are not separated by commas or dots:

$task-$container_name
$task/$container_name
$task_$container_name
$taskfooo$container_name

However, the following are valid:

$task.$container_name
$task.resource.$container_name
$task.fooo.$container_name

And the following are valid since they only contain one template variable with nothing after it:

fooo$task
fooo____$task
fooo/bar$container_name

Metrics Tutorial

Fluent Bit has different input plugins (cpu, mem, disk, netif) to collect host resource usage metrics. cloudwatch_logs output plugin can be used to send these host metrics to CloudWatch in Embedded Metric Format (EMF). If data comes from any of the above mentioned input plugins, cloudwatch_logs output plugin will convert them to EMF format and sent to CloudWatch as JSON log. Additionally, if we set json/emf as the value of log_format config option, CloudWatch will extract custom metrics from embedded JSON payload.

Note: Right now, only cpu and mem metrics can be sent to CloudWatch.

For using the mem input plugin and sending memory usage metrics to CloudWatch, we can consider the following example config file. Here, we use the aws filter which adds ec2_instance_id and az (availability zone) to the log records. Later, in the output config section, we set ec2_instance_id as our metric dimension.

[SERVICE]
    Log_Level info

[INPUT]
    Name mem
    Tag mem

[FILTER]
    Name aws
    Match *

[OUTPUT]
    Name cloudwatch_logs
    Match *
    log_stream_name fluent-bit-cloudwatch
    log_group_name fluent-bit-cloudwatch
    region us-west-2
    log_format json/emf
    metric_namespace fluent-bit-metrics
    metric_dimensions ec2_instance_id
    auto_create_group true

The following config will set two dimensions to all of our metrics- ec2_instance_id and az.

[FILTER]
    Name aws
    Match *

[OUTPUT]
    Name cloudwatch_logs
    Match *
    log_stream_name fluent-bit-cloudwatch
    log_group_name fluent-bit-cloudwatch
    region us-west-2
    log_format json/emf
    metric_namespace fluent-bit-metrics
    metric_dimensions ec2_instance_id,az
    auto_create_group true

AWS for Fluent Bit

Amazon distributes a container image with Fluent Bit and these plugins.

GitHub

github.com/aws/aws-for-fluent-bit

Amazon ECR Public Gallery

aws-for-fluent-bit

Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:<tag>

For example, you can pull the image with latest version by:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

If you see errors for image pull limits, try log into public ECR with your AWS credentials:

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

You can check the Amazon ECR Public official doc for more details

Docker Hub

amazon/aws-for-fluent-bit

Amazon ECR

You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:

aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/

For more see the AWS for Fluent Bit github repo.

Amazon Kinesis Data Firehose

Send logs to Amazon Kinesis Firehose

The Amazon Kinesis Data Firehose output plugin allows to ingest your records into the Firehose service.

This is the documentation for the core Fluent Bit Firehose plugin written in C. It can replace the aws/amazon-kinesis-firehose-for-fluent-bit Golang Fluent Bit plugin released last year. The Golang plugin was named firehose; this new high performance and highly efficient firehose plugin is called kinesis_firehose to prevent conflicts/confusion.

See here for details on how AWS credentials are fetched.

Configuration Parameters

Getting Started

In order to send records into Amazon Kinesis Data Firehose, you can run the plugin from the command line or through the configuration file:

Command Line

The firehose plugin, can read the parameters from the command line through the -p argument (property), e.g:

$ fluent-bit -i cpu -o kinesis_firehose -p delivery_stream=my-stream -p region=us-west-2 -m '*' -f 1

Configuration File

In your main configuration file append the following Output section:

[OUTPUT]
    Name  kinesis_firehose
    Match *
    region us-east-1
    delivery_stream my-stream

Permissions

The following AWS IAM permissions are required to use this plugin:

{
	"Version": "2012-10-17",
	"Statement": [{
		"Effect": "Allow",
		"Action": [
			"firehose:PutRecordBatch"
		],
		"Resource": "*"
	}]
}

Worker support

Fluent Bit 1.7 adds a new feature called workers which enables outputs to have dedicated threads. This kinesis_firehose plugin fully supports workers.

Example:

[OUTPUT]
    Name  kinesis_firehose
    Match *
    region us-east-1
    delivery_stream my-stream
    workers 2

If you enable a single worker, you are enabling a dedicated thread for your Firehose output. We recommend starting with without workers, evaluating the performance, and then adding workers one at a time until you reach your desired/needed throughput. For most users, no workers or a single worker will be sufficient.

AWS for Fluent Bit

Amazon distributes a container image with Fluent Bit and these plugins.

GitHub

github.com/aws/aws-for-fluent-bit

Amazon ECR Public Gallery

aws-for-fluent-bit

Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:<tag>

For example, you can pull the image with latest version by:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

If you see errors for image pull limits, try log into public ECR with your AWS credentials:

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

You can check the Amazon ECR Public official doc for more details.

Docker Hub

amazon/aws-for-fluent-bit

Amazon ECR

You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:

aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/

For more see the AWS for Fluent Bit github repo.

Amazon Kinesis Data Streams

Send logs to Amazon Kinesis Streams

The Amazon Kinesis Data Streams output plugin allows to ingest your records into the service.

This is the documentation for the core Fluent Bit Kinesis plugin written in C. It has all the core features of the Golang Fluent Bit plugin released in 2019. The Golang plugin was named kinesis; this new high performance and highly efficient kinesis plugin is called kinesis_streams to prevent conflicts/confusion.

Currently, this kinesis_streams plugin will always use a random partition key when uploading records to kinesis via the .

Configuration Parameters

Getting Started

In order to send records into Amazon Kinesis Data Streams, you can run the plugin from the command line or through the configuration file:

Command Line

The kinesis_streams plugin, can read the parameters from the command line through the -p argument (property), e.g:

Configuration File

In your main configuration file append the following Output section:

Permissions

The following AWS IAM permissions are required to use this plugin:

AWS for Fluent Bit

Amazon distributes a container image with Fluent Bit and these plugins.

GitHub

Amazon ECR Public Gallery

Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command:

For example, you can pull the image with latest version by:

If you see errors for image pull limits, try log into public ECR with your AWS credentials:

Docker Hub

Amazon ECR

You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:

Amazon S3

Send logs, data, and metrics to Amazon S3

The Amazon S3 output plugin lets you ingest records into the cloud object store.

The plugin can upload data to S3 using the or . Multipart is the default and is recommended. Fluent Bit will stream data in a series of parts. This limits the amount of data buffered on disk at any point in time. By default, every time 5 MiB of data have been received, a new part will be uploaded. The plugin can create files up to gigabytes in size from many small chunks or parts using the multipart API. All aspects of the upload process are configurable.

The plugin lets you specify a maximum file size, and a timeout for uploads. A file will be created in S3 when the maximum size or the timeout is reached, whichever comes first.

Records are stored in files in S3 as newline delimited JSON.

Configuration Parameters

TLS / SSL

Permissions

The plugin requires the following AWS IAM permissions:

Differences between S3 and other Fluent Bit outputs

The S3 output plugin is used to upload large files to an Amazon S3 bucket, while most other outputs which send many requests to upload data in batches of a few megabytes or less.

When Fluent Bit receives logs, it stores them in chunks, either in memory or the filesystem depending on your settings. Chunks are usually around 2 MB in size. Fluent Bit sends chunks, in order, to each output that matches their tag. Most outputs then send the chunk immediately to their destination. A chunk is sent to the output's flush callback function, which must return one of FLB_OK, FLB_RETRY, or FLB_ERROR. Fluent Bit keeps count of the return values from each output's flush callback function. These counters are the data source for Fluent Bit's error, retry, and success metrics available in Prometheus format through its monitoring interface.

The S3 output plugin conforms to the Fluent Bit output plugin specification. Since S3's use case is to upload large files (over 2 MB), its behavior is different. S3's flush callback function buffers the incoming chunk to the filesystem, and returns an FLB_OK. This means Prometheus metrics available from the Fluent Bit HTTP server are meaningless for S3. In addition, the storage.total_limit_size parameter is not meaningful for S3 since it has its own buffering system in the store_dir. Instead, use store_dir_limit_size. S3 requires a writeable filesystem. Running Fluent Bit on a read-only filesystem won't work with the S3 output.

S3 has its own buffering system and its own callback to upload data, so the normal sequential data ordering of chunks provided by the Fluent Bit engine may be compromised. S3 has the presevere_data_ordering option which ensures data is uploaded in the original order it was collected by Fluent Bit.

Summary: Uniqueness in S3 Plugin

You must use store_dir_limit_size to limit the space on disk used by S3 buffer files.
The original ordering of data inputted to Fluent Bit may not be preserved unless you enable preserve_data_ordering On.

S3 Key Format and Tag Delimiters

In Fluent Bit, all logs have an associated tag. The s3_key_format option lets you inject the tag into the S3 key using the following syntax:

$TAG: The full tag.
$TAG[n]: The nth part of the tag (index starting at zero). This syntax is copied from the rewrite tag filter. By default, “parts” of the tag are separated with dots, but you can change this with s3_key_format_tag_delimiters.

In the following example, assume the date is January 1st, 2020 00:00:00 and the tag associated with the logs in question is my_app_name-logs.prod.

With the delimiters as . and -, the tag splits into parts as follows:

$TAG[0] = my_app_name
$TAG[1] = logs
$TAG[2] = prod

The key in S3 will be /prod/my_app_name/2020/01/01/00/00/00/bgdHN1NM.gz.

Allowing a file extension in the S3 Key Format with $UUID

The Fluent Bit S3 output was designed to ensure that previous uploads will never be overwritten by a subsequent upload. The s3_key_format supports time formatters, $UUID, and $INDEX. $INDEX is special because it is saved in the store_dir. If you restart Fluent Bit with the same disk, it can continue incrementing the index from its last value in the previous run.

For files uploaded with the PutObject API, the S3 output requires that a unique random string be present in the S3 key. Many of the use cases for PutObject uploads involve a short time period between uploads, so a timestamp in the S3 key may not be unique enough between uploads. For example, if you only specify minute granularity timestamps in the S3 key, with a small upload size, it is possible to have two uploads that have timestamps set in the same minute. This requirement can be disabled with static_file_path On.

The PutObject API is used in these cases:

When you explicitly set use_put_object On.
On startup when the S3 output finds old buffer files in the store_dir from a previous run and attempts to send all of them at once.
On shutdown. To prevent data loss the S3 output attempts to send all currently buffered data at once.

You should always specify $UUID somewhere in your S3 key format. Otherwise, if the PutObject API is used, S3 appends a random eight-character UUID to the end of your S3 key. This means that a file extension set at the end of an S3 key will have the random UUID appended to it. Disabled this with static_file_path On.

For example, we attempt to set a .gz extension without specifying $UUID:

In the case where pending data is uploaded on shutdown, if the tag was app, the S3 key in the S3 bucket might be:

The S3 output appended a random string to the file extension, since this upload on shutdown used the PutObject API.

There are two ways of disabling this behavior:

Use static_file_path:
Explicitly define where the random UUID will go in the S3 key format:

Reliability

The store_dir is used to temporarily store data before upload. If Fluent Bit stops suddenly, it will try to send all data and complete all uploads before it shuts down. If it can not send some data, on restart it will look in the store_dir for existing data and try to send it.

Multipart uploads are ideal for most use cases because they allow the plugin to upload data in small chunks over time. For example, 1 GB file can be created from 200 5 MB chunks. While the file size in S3 will be 1 GB, only 5 MB will be buffered on disk at any one point in time.

Using S3 without persisted disk

In these situations, we recommend using the PutObject API and sending data frequently, to avoid local buffering as much as possible. This will limit data loss in the event Fluent Bit is killed unexpectedly.

The following settings are recommended for this use case:

S3 Multipart Uploads

Fallback to `PutObject`

Uploads are triggered by these settings:

total_file_size and upload_chunk_size: When S3 has buffered data in the store_dir that meets the desired total_file_size (for use_put_object On) or the upload_chunk_size (for Multipart), it will trigger an upload operation.

If you encounter this frequently, use the numbers in the messages to guess your compression factor. In this example, the buffered data was reduced from 5,630,650 bytes to 1,063,320 bytes. The compressed size is one-fifth the actual data size. Configuring upload_chunk_size 30M should ensure each part is large enough after compression to be over the minimum required part size of 5,242,880 bytes.

The S3 API allows the last part in an upload to be less than the 5,242,880 byte minimum. If a part is too small for an existing upload, the S3 output will upload that part and then complete the upload.

`upload_timeout` constrains total multipart upload time for a single file

Completing uploads

Usage with MinIO

Example:

The records store in the MinIO server.

Get Started

To send records into Amazon S3, you can run the plugin from the command line or through the configuration file.

Command Line

The S3 plugin reads parameters from the command line through the -p argument:

Configuration File

In your main configuration file append the following Output section:

An example using PutObject instead of multipart:

AWS for Fluent Bit

Amazon distributes a container image with Fluent Bit and plugins.

GitHub

Amazon ECR Public Gallery

You can download images with different tags using the following command:

For example, you can pull the image with latest version with:

If you see errors for image pull limits, try signing in to public ECR with your AWS credentials:

Docker Hub

Amazon ECR

Use our SSM Public Parameters to find the Amazon ECR image URI in your region:

Advanced usage

Use Apache Arrow for in-memory data processing

To use this feature, FLB_ARROW must be turned on at compile time. Use the following commands:

After being compiled, Fluent Bit can upload incoming data to S3 in Apache Arrow format.

For example:

Setting Compression to arrow makes Fluent Bit convert payload into Apache Arrow format.

Load, analyze, and process stored data using popular data processing tools such as Python pandas, Apache Spark and Tensorflow.

The following example uses pyarrow to analyze the uploaded data:

Azure Blob

Official and Microsoft Certified Azure Storage Blob connector

The Azure Blob output plugin allows ingesting your records into Azure Blob Storage service. This connector is designed to use the Append Blob and Block Blob API.

Our plugin works with the official Azure Service and also can be configured to be used with a service emulator such as Azurite.

Azure Storage Account

Before getting started, make sure you already have an Azure Storage account. As a reference, the following link explains step-by-step how to set up your account:

Azure Blob Storage Tutorial (Video)

Configuration Parameters

We expose different configuration properties. The following table lists all the options available, and the next section has specific configuration details for the official service or the emulator.

Key

Description

default

Getting Started

As mentioned above, you can either deliver records to the official service or an emulator. Below we have an example for each use case.

Configuration for Azure Storage Service

The following configuration example generates a random message with a custom tag:

[SERVICE]
    flush     1
    log_level info

[INPUT]
    name      dummy
    dummy     {"name": "Fluent Bit", "year": 2020}
    samples   1
    tag       var.log.containers.app-default-96cbdef2340.log

[OUTPUT]
    name                  azure_blob
    match                 *
    account_name          YOUR_ACCOUNT_NAME
    shared_key            YOUR_SHARED_KEY
    path                  kubernetes
    container_name        logs
    auto_create_container on
    tls                   on

After you run the configuration file above, you will be able to query the data using the Azure Storage Explorer. The example above will generate the following content in the explorer:

Configuring and using Azure Emulator: Azurite

Install and run Azurite

The quickest way to get started is to install Azurite using npm:

$ npm install -g azurite

then run the service:

$ azurite
Azurite Blob service is starting at http://127.0.0.1:10000
Azurite Blob service is successfully listening at http://127.0.0.1:10000
Azurite Queue service is starting at http://127.0.0.1:10001
Azurite Queue service is successfully listening at http://127.0.0.1:10001

Configuring Fluent Bit for Azurite

Azurite comes with a default account_name and shared_key, so make sure to use the specific values provided in the example below (do an exact copy/paste):

[SERVICE]
    flush     1
    log_level info

[INPUT]
    name      dummy
    dummy     {"name": "Fluent Bit", "year": 2020}
    samples   1
    tag       var.log.containers.app-default-96cbdef2340.log

[OUTPUT]
    name                  azure_blob
    match                 *
    account_name          devstoreaccount1
    shared_key            Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
    path                  kubernetes
    container_name        logs
    auto_create_container on
    tls                   off
    emulator_mode         on
    endpoint              http://127.0.0.1:10000

after running that Fluent Bit configuration you will see the data flowing into Azurite:

$ azurite
Azurite Blob service is starting at http://127.0.0.1:10000
Azurite Blob service is successfully listening at http://127.0.0.1:10000
Azurite Queue service is starting at http://127.0.0.1:10001
Azurite Queue service is successfully listening at http://127.0.0.1:10001
127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "GET /devstoreaccount1/logs?restype=container HTTP/1.1" 404 -
127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs?restype=container HTTP/1.1" 201 -
127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log?comp=appendblock HTTP/1.1" 404 -
127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log HTTP/1.1" 201 -
127.0.0.1 - - [03/Sep/2020:17:40:04 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log?comp=appendblock HTTP/1.1" 201 -

Azure Data Explorer

Send logs to Azure Data Explorer (Kusto)

The Kusto output plugin allows to ingest your logs into an Azure Data Explorer cluster, via the Queued Ingestion mechanism. This output plugin can also be used to ingest logs into an Eventhouse cluster in Microsoft Fabric Real Time Analytics.

For ingesting into Azure Data Explorer: Creating a Kusto Cluster and Database

You can create an Azure Data Explorer cluster in one of the following ways:

For ingesting into Microsoft Fabric Real Time Analytics : Creating an Eventhouse Cluster and KQL Database

You can create an Eventhouse cluster and a KQL database follow the following steps:

Creating an Azure Registered Application

Fluent-Bit will use the application's credentials, to ingest data into your cluster.

Creating a Table

Fluent-Bit ingests the event data into Kusto in a JSON format, that by default will include 3 properties:

log - the actual event payload.
tag - the event tag.
timestamp - the event timestamp.

A table with the expected schema must exist in order for data to be ingested properly.

.create table FluentBit (log:dynamic, tag:string, timestamp:datetime)

Optional - Creating an Ingestion Mapping

By default, Kusto will insert incoming ingestions into a table by inferring the mapped table columns, from the payload properties. However, this mapping can be customized by creatng a JSON ingestion mapping. The plugin can be configured to use an ingestion mapping via the ingestion_mapping_reference configuration key.

Configuration Parameters

Configuration File

Get started quickly with this configuration file:

[OUTPUT]
    Match *
    Name azure_kusto
    Tenant_Id <app_tenant_id>
    Client_Id <app_client_id>
    Client_Secret <app_secret>
    Ingestion_Endpoint https://ingest-<cluster>.<region>.kusto.windows.net
    Database_Name <database_name>
    Table_Name <table_name>
    Ingestion_Mapping_Reference <mapping_name>

Troubleshooting

403 Forbidden

If you get a 403 Forbidden error response, make sure that:

You provided the correct AAD registered application credentials.
You authorized the application to ingest into your database or table.

Azure Log Analytics

Send logs, metrics to Azure Log Analytics

Azure output plugin allows to ingest your records into Azure Log Analytics service.

To get more details about how to setup Azure Log Analytics, please refer to the following documentation: Azure Log Analytics

Configuration Parameters

Getting Started

In order to insert records into an Azure Log Analytics instance, you can run the plugin from the command line or through the configuration file:

Command Line

The azure plugin, can read the parameters from the command line in two ways, through the -p argument (property), e.g:

$ fluent-bit -i cpu -o azure -p customer_id=abc -p shared_key=def -m '*' -f 1

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name  cpu

[OUTPUT]
    Name        azure
    Match       *
    Customer_ID abc
    Shared_Key  def

Another example using the Log_Type_Key with record-accessor, which will read the table name (or event type) dynamically from kubernetes label app, instead of Log_Type:

[INPUT]
    Name  cpu

[OUTPUT]
    Name        azure
    Match       *
    Log_Type_Key $kubernetes['labels']['app']
    Customer_ID abc
    Shared_Key  def

Azure Logs Ingestion API

Send logs to Azure Log Analytics using Logs Ingestion API with DCE and DCR

Azure Logs Ingestion plugin allows you ingest your records using to supported or to that you create.

The Logs ingestion API requires the following components:

A Data Collection Endpoint (DCE)
A Data Collection Rule (DCR) and
A Log Analytics Workspace

To get more details about how to setup these components, please refer to the following documentations:

Configuration Parameters

Getting Started

To send records into an Azure Log Analytics using Logs Ingestion API the following resources needs to be created:

A Data Collection Endpoint (DCE) for ingestion
A Data Collection Rule (DCR) for data transformation
An app registration with client secrets (for DCR access).

Configuration File

Use this configuration to quickly get started:

Setup your DCR transformation accordingly based on the json output from fluent-bit's pipeline (input, parser, filter, output).

Counter

Counter is a very simple plugin that counts how many records it's getting upon flush time. Plugin output is as follows:

Getting Started

You can run the plugin from the command line or through the configuration file:

Command Line

From the command line you can let Fluent Bit count up a data with the following options:

Configuration File

In your main configuration file append the following Input & Output sections:

Testing

Once Fluent Bit is running, you will see the reports in the output interface similar to this:

Datadog

Send logs to Datadog

The Datadog output plugin allows to ingest your logs into .

Before you begin, you need a , a , and you need to .

Configuration Parameters

Key

Description

Default

Configuration File

Get started quickly with this configuration file:

Troubleshooting

403 Forbidden

Elasticsearch

Send logs to Elasticsearch (including Amazon OpenSearch Service)

The es output plugin, allows to ingest your records into an database. The following instructions assumes that you have a fully operational Elasticsearch service running in your environment.

Configuration Parameters

Key

Description

default

TLS / SSL

write_operation

The write_operation can be any of:

Please note, Id_Key or Generate_ID is required in update, and upsert scenario.

Getting Started

In order to insert records into a Elasticsearch service, you can run the plugin from the command line or through the configuration file:

Command Line

The es plugin, can read the parameters from the command line in two ways, through the -p argument (property) or setting them directly through the service URI. The URI format is the following:

Using the format specified, you could start Fluent Bit through:

which is similar to do:

Configuration File

About Elasticsearch field names

Some input plugins may generate messages where the field names contains dots, since Elasticsearch 2.0 this is not longer allowed, so the current es plugin replaces them with an underscore, e.g:

becomes

FAQ

Elasticsearch rejects requests saying "the final mapping would have more than 1 type"

Since Elasticsearch 6.0, you cannot create multiple types in a single index. This means that you cannot set up your configuration as below anymore.

If you see an error message like below, you'll need to fix your configuration to use a single type on each index.

Rejecting mapping update to [search] as the final mapping would have more than 1 type

Elasticsearch rejects requests saying "Document mapping type name can't start with '_'"

Fluent Bit + Amazon OpenSearch Service

The Amazon OpenSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon OpenSearch Service with IAM Authentication.

Example configuration:

Notice that the Port is set to 443, tls is enabled, and AWS_Region is set.

Fluent Bit + Elastic Cloud

Example configuration:

Validation Failed: 1: an id must be provided if version type or value are set

Since v1.8.2, Fluent Bit started using create method (instead of index) for data submission. This makes Fluent Bit compatible with Datastream introduced in Elasticsearch 7.9.

If you see action_request_validation_exception errors on your pipeline with Fluent Bit >= v1.8.2, you can fix it up by turning on Generate_ID as follows:

Action/metadata contains an unknown parameter type

Elastic Cloud is now on version 8 so the type option must be removed by setting Suppress_Type_Name On as indicated above.

Without this you will see errors like:

Logstash_Prefix_Key

The following snippet demonstrates using the namespace name as extracted by the kubernetes filter as logstash prefix:

For records that do nor have the field kubernetes.namespace_name, the default prefix, logstash will be used.

Amazon CloudWatch

Send logs and metrics to Amazon CloudWatch

The Amazon CloudWatch output plugin allows to ingest your records into the CloudWatch Logs service. Support for CloudWatch Metrics is also provided via EMF.

See here for details on how AWS credentials are fetched.

Configuration Parameters

Key

Description

Getting Started

In order to send records into Amazon Cloudwatch, you can run the plugin from the command line or through the configuration file:

Command Line

The cloudwatch plugin, can read the parameters from the command line through the -p argument (property), e.g:

$ fluent-bit -i cpu -o cloudwatch_logs -p log_group_name=group -p log_stream_name=stream -p region=us-west-2 -m '*' -f 1

Configuration File

In your main configuration file append the following Output section:

[OUTPUT]
    Name cloudwatch_logs
    Match   *
    region us-east-1
    log_group_name fluent-bit-cloudwatch
    log_stream_prefix from-fluent-bit-
    auto_create_group On

Intergration with Localstack (Cloudwatch Logs)

For an instance of Localstack running at http://localhost:4566, the following configuration needs to be added to the [OUTPUT] section:

endpoint localhost
port 4566

Any testing credentials can be exported as local variables, such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Permissions

The following AWS IAM permissions are required to use this plugin:

{
	"Version": "2012-10-17",
	"Statement": [{
		"Effect": "Allow",
		"Action": [
			"logs:CreateLogStream",
			"logs:CreateLogGroup",
			"logs:PutLogEvents"
		],
		"Resource": "*"
	}]
}

Log Stream and Group Name templating using record_accessor syntax

Here is an example usage, for a common use case- templating log group and stream names based on Kubernetes metadata.

Recall that the kubernetes filter can add metadata which will look like the following:

kubernetes: {
    annotations: {
        "kubernetes.io/psp": "eks.privileged"
    },
    container_hash: "<some hash>",
    container_name: "myapp",
    docker_id: "<some id>",
    host: "ip-10-1-128-166.us-east-2.compute.internal",
    labels: {
        app: "myapp",
        "pod-template-hash": "<some hash>"
    },
    namespace_name: "my-namespace",
    pod_id: "198f7dd2-2270-11ea-be47-0a5d932f5920",
    pod_name: "myapp-5468c5d4d7-n2swr"
}

Using record_accessor, we can build a template based on this object.

Here is our output configuration:

[OUTPUT]
    Name cloudwatch_logs
    Match   *
    region us-east-1
    log_group_name fallback-group
    log_stream_prefix fallback-stream
    auto_create_group On
    log_group_template application-logs-$kubernetes['host'].$kubernetes['namespace_name']
    log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']

If the kubernetes structure is not found in the log record, then the log_group_name and log_stream_prefix will be used instead, and Fluent Bit will log an error like:

[2022/06/30 06:09:29] [ warn] [record accessor] translation failed, root key=kubernetes

Limitations of record_accessor syntax

Assume that your log records contain the metadata keys container_name and task. The following would be invalid templates because the two template variables are not separated by commas or dots:

$task-$container_name
$task/$container_name
$task_$container_name
$taskfooo$container_name

However, the following are valid:

$task.$container_name
$task.resource.$container_name
$task.fooo.$container_name

And the following are valid since they only contain one template variable with nothing after it:

fooo$task
fooo____$task
fooo/bar$container_name

Metrics Tutorial

Note: Right now, only cpu and mem metrics can be sent to CloudWatch.

[SERVICE]
    Log_Level info

[INPUT]
    Name mem
    Tag mem

[FILTER]
    Name aws
    Match *

[OUTPUT]
    Name cloudwatch_logs
    Match *
    log_stream_name fluent-bit-cloudwatch
    log_group_name fluent-bit-cloudwatch
    region us-west-2
    log_format json/emf
    metric_namespace fluent-bit-metrics
    metric_dimensions ec2_instance_id
    auto_create_group true

The following config will set two dimensions to all of our metrics- ec2_instance_id and az.

[FILTER]
    Name aws
    Match *

[OUTPUT]
    Name cloudwatch_logs
    Match *
    log_stream_name fluent-bit-cloudwatch
    log_group_name fluent-bit-cloudwatch
    region us-west-2
    log_format json/emf
    metric_namespace fluent-bit-metrics
    metric_dimensions ec2_instance_id,az
    auto_create_group true

AWS for Fluent Bit

Amazon distributes a container image with Fluent Bit and these plugins.

GitHub

github.com/aws/aws-for-fluent-bit

Amazon ECR Public Gallery

aws-for-fluent-bit

Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:<tag>

For example, you can pull the image with latest version by:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

If you see errors for image pull limits, try log into public ECR with your AWS credentials:

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

You can check the Amazon ECR Public official doc for more details

Docker Hub

amazon/aws-for-fluent-bit

Amazon ECR

You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:

aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/

For more see the AWS for Fluent Bit github repo.

Amazon S3

Send logs, data, and metrics to Amazon S3

The Amazon S3 output plugin lets you ingest records into the cloud object store.

The plugin lets you specify a maximum file size, and a timeout for uploads. A file will be created in S3 when the maximum size or the timeout is reached, whichever comes first.

Records are stored in files in S3 as newline delimited JSON.

See for details about fetching AWS credentials.

The output by the built-in http server in Fluent Bit are meaningless for S3 output. S3 has its own buffering and retry mechanisms. The Fluent Bit AWS S3 maintainers apologize for this feature gap; you can .

Configuration Parameters

Key

Description

Default

TLS / SSL

To skip TLS verification, set tls.verify as false. For more details about the properties available and general configuration, refer to .

Permissions

The plugin requires the following AWS IAM permissions:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "s3:PutObject"
        ],
        "Resource": "*"
    }]
}

Differences between S3 and other Fluent Bit outputs

The S3 output plugin is used to upload large files to an Amazon S3 bucket, while most other outputs which send many requests to upload data in batches of a few megabytes or less.

S3 uploads primarily initiate using the S3 callback function, which runs separately from its flush.

Summary: Uniqueness in S3 Plugin

The HTTP Monitoring interface output metrics are not meaningful for S3. AWS understands that this is non-ideal; we have to allow S3 to manage its own output metrics.
You must use store_dir_limit_size to limit the space on disk used by S3 buffer files.
The original ordering of data inputted to Fluent Bit may not be preserved unless you enable preserve_data_ordering On.

S3 Key Format and Tag Delimiters

In Fluent Bit, all logs have an associated tag. The s3_key_format option lets you inject the tag into the S3 key using the following syntax:

$TAG: The full tag.
$TAG[n]: The nth part of the tag (index starting at zero). This syntax is copied from the rewrite tag filter. By default, “parts” of the tag are separated with dots, but you can change this with s3_key_format_tag_delimiters.

In the following example, assume the date is January 1st, 2020 00:00:00 and the tag associated with the logs in question is my_app_name-logs.prod.

With the delimiters as . and -, the tag splits into parts as follows:

$TAG[0] = my_app_name
$TAG[1] = logs
$TAG[2] = prod

The key in S3 will be /prod/my_app_name/2020/01/01/00/00/00/bgdHN1NM.gz.

Allowing a file extension in the S3 Key Format with $UUID

The PutObject API is used in these cases:

When you explicitly set use_put_object On.
On startup when the S3 output finds old buffer files in the store_dir from a previous run and attempts to send all of them at once.
On shutdown. To prevent data loss the S3 output attempts to send all currently buffered data at once.

For example, we attempt to set a .gz extension without specifying $UUID:

In the case where pending data is uploaded on shutdown, if the tag was app, the S3 key in the S3 bucket might be:

/app/2022/12/25/00_00_00.gz-apwgylqg

The S3 output appended a random string to the file extension, since this upload on shutdown used the PutObject API.

There are two ways of disabling this behavior:

Use static_file_path:
Explicitly define where the random UUID will go in the S3 key format:

Reliability

One drawback to multipart uploads is that the file and data aren't visible in S3 until the upload is completed with a call. The plugin attempts to make this call whenever Fluent Bit is shut down to ensure your data is available in S3. It also stores metadata about each upload in the store_dir, ensuring that uploads can be completed when Fluent Bit restarts (assuming it has access to persistent disk and the store_dir files will still be present on restart).

Using S3 without persisted disk

If you run Fluent Bit in an environment without persistent disk, or without the ability to restart Fluent Bit and give it access to the data stored in the store_dir from previous executions, some considerations apply. This might occur if you run Fluent Bit on .

The following settings are recommended for this use case:

S3 Multipart Uploads

With use_put_object Off (default), S3 will attempt to send files using multipart uploads. For each file, S3 first calls , then a series of calls to for each fragment (targeted to be upload_chunk_size bytes), and finally to create the final file in S3.

Fallback to `PutObject`

S3 each fragment to be at least 5,242,880 bytes, otherwise the upload is rejected.

The S3 output must sometimes fallback to the .

Uploads are triggered by these settings:

total_file_size and upload_chunk_size: When S3 has buffered data in the store_dir that meets the desired total_file_size (for use_put_object On) or the upload_chunk_size (for Multipart), it will trigger an upload operation.
upload_timeout: Whenever locally buffered data has been present on the filesystem in the store_dir longer than the configured upload_timeout, it will be sent even when the desired byte size hasn't been reached. If you configure a small upload_timeout, your files may be smaller than the total_file_size. The timeout is evaluated against the time at which S3 started buffering data for each unqiue tag (that is, the time when new data was buffered for the unique tag after the last upload). The timeout is also evaluated against the time, so a multipart upload will be completed after upload_timeout has elapsed, even if the desired size has not yet been reached.

If your upload_timeout triggers an upload before the pending buffered data reaches the upload_chunk_size, it may be too small for a multipart upload. S3 will fallback to use the .

When you enable compression, S3 applies the compression algorithm at send time. The size settings trigger uploads based on the size of buffered data, not the final compressed size. It's possible that after compression, buffered data no longer meets the required minimum S3 size. If this occurs, you will see a log message like:

[ info] [output:s3:s3.0] Pre-compression upload_chunk_size= 5630650, After
compression, chunk is only 1063320 bytes, the chunk was too small, using PutObject to upload

`upload_timeout` constrains total multipart upload time for a single file

The upload_timeout evaluated against the time. A multipart upload will be completed after upload_timeout elapses, even if the desired size has not yet been reached.

Completing uploads

When is called, an UploadID is returned. S3 stores these IDs for active uploads in the store_dir. Until is called, the uploaded data isn't visible in S3.

On shutdown, S3 output attempts to complete all pending uploads. If an upload fails to complete, the ID remains buffered in the store_dir in a directory called multipart_upload_metadata. If you restart the S3 output with the same store_dir it will discover the old UploadIDs and complete the pending uploads. The has suggestions on discovering and deleting or completing dangling uploads in your buckets.

Usage with MinIO

is a high-performance, S3 compatible object storage and you can build your app with S3 functionality without S3.

The following example runs at localhost:9000, and create a bucket of your-bucket.

Example:

The records store in the MinIO server.

Get Started

To send records into Amazon S3, you can run the plugin from the command line or through the configuration file.

Command Line

The S3 plugin reads parameters from the command line through the -p argument:

fluent-bit -i cpu -o s3 -p bucket=my-bucket -p region=us-west-2 -p -m '*' -f 1

Configuration File

In your main configuration file append the following Output section:

An example using PutObject instead of multipart:

AWS for Fluent Bit

Amazon distributes a container image with Fluent Bit and plugins.

GitHub

Amazon ECR Public Gallery

Our images are available in the Amazon ECR Public Gallery as .

You can download images with different tags using the following command:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:<tag>

For example, you can pull the image with latest version with:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

If you see errors for image pull limits, try signing in to public ECR with your AWS credentials:

See the for more details.

Docker Hub

is also available from the Docker Hub.

Amazon ECR

Use our SSM Public Parameters to find the Amazon ECR image URI in your region:

For more information, see the .

Advanced usage

Use Apache Arrow for in-memory data processing

With Fluent Bit v1.8 or greater, the Amazon S3 plugin includes the support for . Support isn't enabled by default, and has a dependency on a shared version of libarrow.

To use this feature, FLB_ARROW must be turned on at compile time. Use the following commands:

After being compiled, Fluent Bit can upload incoming data to S3 in Apache Arrow format.

For example:

Setting Compression to arrow makes Fluent Bit convert payload into Apache Arrow format.

Load, analyze, and process stored data using popular data processing tools such as Python pandas, Apache Spark and Tensorflow.

The following example uses pyarrow to analyze the uploaded data:

Loki

Loki is multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate.

The Fluent Bit loki built-in output plugin allows you to send your log or events to a Loki service. It supports data enrichment with Kubernetes labels, custom label keys and Tenant ID within others.

Be aware there is a separate Golang output plugin provided by Grafana with different configuration options.

Configuration Parameters

Key

Description

Default

Labels

Loki store the record logs inside Streams, a stream is defined by a set of labels, at least one label is required.

Fluent Bit implements a flexible mechanism to set labels by using fixed key/value pairs of text but also allowing to set as labels certain keys that exists as part of the records that are being processed. Consider the following JSON record (pretty printed for readability):

{
    "key": 1,
    "sub": {
        "stream": "stdout",
        "id": "some id"
    },
    "kubernetes": {
        "labels": {
            "team": "Santiago Wanderers"
        }
    }
}

If you decide that your Loki Stream will be composed by two labels called job and the value of the record key called stream , your labels configuration properties might look as follows:

[OUTPUT]
    name   loki
    match  *
    labels job=fluentbit, $sub['stream']

As you can see the label job has the value fluentbit and the second label is configured to access the nested map called sub targeting the value of the key stream . Note that the second label name must starts with a $, that means that's a Record Accessor pattern so it provide you the ability to retrieve values from nested maps by using the key names.

When processing above's configuration, internally the ending labels for the stream in question becomes:

job="fluentbit", stream="stdout"

Another feature of Labels management is the ability to provide custom key names, using the same record accessor pattern we can specify the key name manually and let the value to be populated automatically at runtime, e.g:

[OUTPUT]
    name   loki
    match  *
    labels job=fluentbit, mystream=$sub['stream']

When processing that new configuration, the internal labels will be:

job="fluentbit", mystream="stdout"

Using the `label_keys` property

The additional configuration property called label_keys allow to specify multiple record keys that needs to be placed as part of the outgoing Stream Labels, yes, this is a similar feature than the one explained above in the labels property. Consider this as another way to set a record key in the Stream, but with the limitation that you cannot use a custom name for the key value.

The following configuration examples generate the same Stream Labels:

[OUTPUT]
    name       loki
    match      *
    labels     job=fluentbit
    label_keys $sub['stream']

the above configuration accomplish the same than this one:

[OUTPUT]
    name   loki
    match  *
    labels job=fluentbit, $sub['stream']

both will generate the following Streams label:

job="fluentbit", stream="stdout"

Using the `label_map_path` property

The configuration property label_map_path is to read a JSON file that defines how to extract labels from each record.

The file should contain a JSON object. Each keys define how to get label value from a nested record. Each values are used as label names.

The following configuration examples generate the same Stream Labels:

map.json:

{
    "sub": {
           "stream": "stream"
    }
}

The following configuration examples generate the same Stream Labels:

[OUTPUT]
    name   loki
    match  *
    label_map_path /path/to/map.json

the above configuration accomplish the same than this one:

[OUTPUT]
    name   loki
    match  *
    labels job=fluentbit, $sub['stream']

both will generate the following Streams label:

job="fluentbit", stream="stdout"

Kubernetes & Labels

Note that if you are running in a Kubernetes environment, you might want to enable the option auto_kubernetes_labels which will auto-populate the streams with the Pod labels for you. Consider the following configuration:

[OUTPUT]
    name                   loki
    match                  *
    labels                 job=fluentbit
    auto_kubernetes_labels on

Based in the JSON example provided above, the internal stream labels will be:

job="fluentbit", team="Santiago Wanderers"

Drop Single Key

If there is only one key remaining after removing keys, you can use the drop_single_key property to send its value to Loki, rather than a single key=value pair.

Consider this simple JSON example:

{"key":"value"}

If the value is a string, line_format is json, and drop_single_key is true, it will be sent as a quoted string.

[OUTPUT]
    name            loki
    match           *
    drop_single_key on
    line_format     json

The outputted line would show in Loki as:

"value"

If drop_single_key is raw, or line_format is key_value, it will show in Loki as:

value

If you want both structured JSON and plain-text logs in Loki, you should set drop_single_key to raw and line_format to json. Loki does not interpret a quoted string as valid JSON, and so to remove the quotes without drop_single_key set to raw, you would need to use a query like this:

{"job"="fluent-bit"} | regexp `^"?(?P<log>.*?)"?$` | line_format "{{.log}}"

If drop_single_key is off, it will show in Loki as:

{"key":"value"}

You can get the same behavior this flag provides in Loki with drop_single_key set to off with this query:

{"job"="fluent-bit"} | json | line_format "{{.log}}"

Structured metadata

Structured metadata lets you attach custom fields to individual log lines without embedding the information in the content of the log line. This capability works well for high cardinality data that isn't suited for using labels. While not a label, the structured_metadata configuration parameter operates similarly to the labels parameter. Both parameters are comma-delimited key=value lists, and both can use record accessors to reference keys within the record being processed.

The following configuration:

Defines fixed values for the cluster and region labels.
Uses the record accessor pattern to set the namespace label to the namespace name as determined by the Kubernetes metadata filter (not shown).
Uses a structured metadata field to hold the Kubernetes pod name.

[OUTPUT]
    name                loki
    match               *
    labels              cluster=my-k8s-cluster, region=us-east-1, namespace=$kubernetes['namespace_name']
    structured_metadata pod=$kubernetes['pod_name']

Other common uses for structured metadata include trace and span IDs, process and thread IDs, and log levels.

Structured metadata is officially supported starting with Loki 3.0, and shouldn't be used with Loki deployments prior to Loki 3.0.

Networking and TLS Configuration

This plugin inherit core Fluent Bit features to customize the network behavior and optionally enable TLS in the communication channel. For more details about the specific options available refer to the following articles:

Networking Setup: timeouts, keepalive and source address
Security & TLS: all about TLS configuration and certificates

Note that all options mentioned in the articles above must be enabled in the plugin configuration in question.

Fluent Bit + Grafana Cloud

Fluent Bit supports sending logs (and metrics) to Grafana Cloud by providing the appropriate URL and ensuring TLS is enabled.

An example configuration - make sure to set the credentials and ensure the host URL matches the correct one for your deployment:

    [OUTPUT]
        Name        loki
        Match       *
        Host        logs-prod-eu-west-0.grafana.net
        port        443
        tls         on
        tls.verify  on
        http_user   XXX
        http_passwd XXX

Getting Started

The following configuration example, will emit a dummy example record and ingest it on Loki . Copy and paste the following content into a file called out_loki.conf:

[SERVICE]
    flush     1
    log_level info

[INPUT]
    name      dummy
    dummy     {"key": 1, "sub": {"stream": "stdout", "id": "some id"}, "kubernetes": {"labels": {"team": "Santiago Wanderers"}}}
    samples   1

[OUTPUT]
    name                   loki
    match                  *
    host                   127.0.0.1
    port                   3100
    labels                 job=fluentbit
    label_keys             $sub['stream']
    auto_kubernetes_labels on

run Fluent Bit with the new configuration file:

$ fluent-bit -c out_loki.conf

Fluent Bit output:

Fluent Bit v1.7.0
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/10/14 20:57:45] [ info] [engine] started (pid=809736)
[2020/10/14 20:57:45] [ info] [storage] version=1.0.6, initializing...
[2020/10/14 20:57:45] [ info] [storage] in-memory
[2020/10/14 20:57:45] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/10/14 20:57:45] [ info] [output:loki:loki.0] configured, hostname=127.0.0.1:3100
[2020/10/14 20:57:45] [ info] [sp] stream processor started
[2020/10/14 20:57:46] [debug] [http] request payload (272 bytes)
[2020/10/14 20:57:46] [ info] [output:loki:loki.0] 127.0.0.1:3100, HTTP status=204

PostgreSQL

PostgreSQL is a very popular and versatile open source database management system that supports the SQL language and that is capable of storing both structured and unstructured data, such as JSON objects.

Given that Fluent Bit is designed to work with JSON objects, the pgsql output plugin allows users to send their data to a PostgreSQL database and store it using the JSONB type.

PostgreSQL 9.4 or higher is required.

Preliminary steps

According to the parameters you have set in the configuration file, the plugin will create the table defined by the table option in the database defined by the database option hosted on the server defined by the host option. It will use the PostgreSQL user defined by the user option, which needs to have the right privileges to create such a table in that database.

NOTE: If you are not familiar with how PostgreSQL's users and grants system works, you might find useful reading the recommended links in the "References" section at the bottom.

A typical installation normally consists of a self-contained database for Fluent Bit in which you can store the output of one or more pipelines. Ultimately, it is your choice to to store them in the same table, or in separate tables, or even in separate databases based on several factors, including workload, scalability, data protection and security.

In this example, for the sake of simplicity, we use a single table called fluentbit in a database called fluentbit that is owned by the user fluentbit. Feel free to use different names. Preferably, for security reasons, do not use the postgres user (which has SUPERUSER privileges).

Create the `fluentbit` user

Generate a robust random password (e.g. pwgen 20 1) and store it safely. Then, as postgres system user on the server where PostgreSQL is installed, execute:

createuser -P fluentbit

At the prompt, please provide the password that you previously generated.

As a result, the user fluentbit without superuser privileges will be created.

If you prefer, instead of the createuser application, you can directly use the SQL command CREATE USER.

Create the `fluentbit` database

As postgres system user, please run:

createdb -O fluentbit fluentbit

This will create a database called fluentbit owned by the fluentbit user. As a result, the fluentbit user will be able to safely create the data table.

Alternatively, you can use the SQL command CREATE DATABASE.

Connection

Make sure that the fluentbit user can connect to the fluentbit database on the specified target host. This might require you to properly configure the pg_hba.conf file.

Configuration Parameters

Libpq

Fluent Bit relies on libpq, the PostgreSQL native client API, written in C language. For this reason, default values might be affected by environment variables and compilation settings. The above table, in brackets, list the most common default values for each connection option.

For security reasons, it is advised to follow the directives included in the password file section.

Configuration Example

In your main configuration file add the following section:

[OUTPUT]
    Name                pgsql
    Match               *
    Host                172.17.0.2
    Port                5432
    User                fluentbit
    Password            YourCrazySecurePassword
    Database            fluentbit
    Table               fluentbit
    Connection_Options  -c statement_timeout=0
    Timestamp_Key       ts

The output table

The output plugin automatically creates a table with the name specified by the table configuration option and made up of the following fields:

tag TEXT
time TIMESTAMP WITHOUT TIMEZONE
data JSONB

As you can see, the timestamp does not contain any information about the time zone and it is therefore referred to the time zone used by the connection to PostgreSQL (timezone setting).

For more information on the JSONB data type in PostgreSQL, please refer to the JSON types page in the official documentation, where you can find instructions on how to index or query the objects (including jsonpath introduced in PostgreSQL 12).

Scalability

PostgreSQL 10 introduces support for declarative partitioning. In order to improve vertical scalability of the database, you can decide to partition your tables on time ranges (for example on a monthly basis). PostgreSQL supports also subpartitions, allowing you to even partition by hash your records (version 11+), and default partitions (version 11+).

For more information on horizontal partitioning in PostgreSQL, please refer to the Table partitioning page in the official documentation.

If you are starting now, our recommendation at the moment is to choose the latest major version of PostgreSQL.

There's more ...

PostgreSQL is a really powerful and extensible database engine. More expert users can indeed take advantage of BEFORE INSERT triggers on the main table and re-route records on normalised tables, depending on tags and content of the actual JSON objects.

For example, you can use Fluent Bit to send HTTP log records to the landing table defined in the configuration file. This table contains a BEFORE INSERT trigger (a function in plpgsql language) that normalises the content of the JSON object and that inserts the record in another table (with its own structure and partitioning model). This kind of triggers allow you to discard the record from the landing table by returning NULL.

References

Here follows a list of useful resources from the PostgreSQL documentation:

Outputs

Amazon CloudWatch

Configuration Parameters

Getting Started

Command Line

Configuration File

Intergration with Localstack (Cloudwatch Logs)

Permissions

Log Stream and Group Name templating using record_accessor syntax

Limitations of record_accessor syntax

Metrics Tutorial

AWS for Fluent Bit

GitHub

Amazon ECR Public Gallery

Docker Hub

Amazon ECR

Amazon Kinesis Data Firehose

Configuration Parameters

Getting Started

Command Line

Configuration File

Permissions

Worker support

AWS for Fluent Bit

GitHub

Amazon ECR Public Gallery

Docker Hub

Amazon ECR

Amazon Kinesis Data Streams

Configuration Parameters

Getting Started

Command Line

Configuration File

Permissions

AWS for Fluent Bit

GitHub

Amazon ECR Public Gallery

Docker Hub

Amazon ECR

Amazon S3

Configuration Parameters

TLS / SSL

Permissions

Differences between S3 and other Fluent Bit outputs

Summary: Uniqueness in S3 Plugin

S3 Key Format and Tag Delimiters

Allowing a file extension in the S3 Key Format with $UUID

Reliability

Using S3 without persisted disk

S3 Multipart Uploads

Fallback to PutObject

upload_timeout constrains total multipart upload time for a single file

Completing uploads

Usage with MinIO

Get Started

Command Line

Configuration File

AWS for Fluent Bit

GitHub

Amazon ECR Public Gallery

Docker Hub

Amazon ECR

Advanced usage

Use Apache Arrow for in-memory data processing

Azure Blob

Azure Storage Account

Configuration Parameters

Getting Started

Configuration for Azure Storage Service

Configuring and using Azure Emulator: Azurite

Install and run Azurite

Configuring Fluent Bit for Azurite

Azure Data Explorer

For ingesting into Azure Data Explorer: Creating a Kusto Cluster and Database

For ingesting into Microsoft Fabric Real Time Analytics : Creating an Eventhouse Cluster and KQL Database

Creating an Azure Registered Application

Creating a Table

Optional - Creating an Ingestion Mapping

Configuration Parameters

Configuration File

Fallback to `PutObject`

`upload_timeout` constrains total multipart upload time for a single file