1 of 44

Outputs

Azure Data Explorer

Send logs to Azure Data Explorer (Kusto)

The Kusto output plugin allows to ingest your logs into an Azure Data Explorer cluster, via the Queued Ingestion mechanism.

Creating a Kusto Cluster and Database

You can create an Azure Data Explorer cluster in one of the following ways:

Creating an Azure Registered Application

Fluent-Bit will use the application's credentials, to ingest data into your cluster.

Creating a Table

Fluent-Bit ingests the event data into Kusto in a JSON format, that by default will include 3 properties:

log - the actual event payload.
tag - the event tag.
timestamp - the event timestamp.

A table with the expected schema must exist in order for data to be ingested properly.

.create table FluentBit (log:dynamic, tag:string, timestamp:datetime)

Optional - Creating an Ingestion Mapping

By default, Kusto will insert incoming ingestions into a table by inferring the mapped table columns, from the payload properties. However, this mapping can be customized by creatng a JSON ingestion mapping. The plugin can be configured to use an ingestion mapping via the ingestion_mapping_reference configuration key.

Configuration Parameters

Key

Description

Default

tenant_id

Required - The tenant/domain ID of the AAD registered application.

client_id

Required - The client ID of the AAD registered application.

client_secret

ingestion_endpoint

Required - The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net

database_name

Required - The database name.

table_name

Required - The table name.

ingestion_mapping_reference

log_key

Key name of the log content.

log

include_tag_key

If enabled, a tag is appended to output. The key name is used tag_key property.

On

tag_key

The key name of tag. If include_tag_key is false, This property is ignored.

tag

include_time_key

If enabled, a timestamp is appended to output. The key name is used time_key property.

On

time_key

The key name of time. If include_time_key is false, This property is ignored.

timestamp

Configuration File

Get started quickly with this configuration file:

[OUTPUT]
    Match *
    Name azure_kusto
    Tenant_Id <app_tenant_id>
    Client_Id <app_client_id>
    Client_Secret <app_secret>
    Ingestion_Endpoint https://ingest-<cluster>.<region>.kusto.windows.net
    Database_Name <database_name>
    Table_Name <table_name>
    Ingestion_Mapping_Reference <mapping_name>

Troubleshooting

403 Forbidden

If you get a 403 Forbidden error response, make sure that:

You provided the correct AAD registered application credentials.
You authorized the application to ingest into your database or table.

Azure Log Analytics

Send logs, metrics to Azure Log Analytics

Azure output plugin allows to ingest your records into service.

To get more details about how to setup Azure Log Analytics, please refer to the following documentation:

Configuration Parameters

Getting Started

In order to insert records into an Azure Log Analytics instance, you can run the plugin from the command line or through the configuration file:

Command Line

The azure plugin, can read the parameters from the command line in two ways, through the -p argument (property), e.g:

Configuration File

In your main configuration file append the following Input & Output sections:

Azure Data Explorer

Send logs to Azure Data Explorer (Kusto)

The Kusto output plugin allows to ingest your logs into an Azure Data Explorer cluster, via the Queued Ingestion mechanism.

Creating a Kusto Cluster and Database

You can create an Azure Data Explorer cluster in one of the following ways:

Creating an Azure Registered Application

Fluent-Bit will use the application's credentials, to ingest data into your cluster.

Creating a Table

Fluent-Bit ingests the event data into Kusto in a JSON format, that by default will include 3 properties:

log - the actual event payload.
tag - the event tag.
timestamp - the event timestamp.

A table with the expected schema must exist in order for data to be ingested properly.

.create table FluentBit (log:dynamic, tag:string, timestamp:datetime)

Optional - Creating an Ingestion Mapping

Configuration Parameters

Key

Description

Default

tenant_id

Required - The tenant/domain ID of the AAD registered application.

client_id

Required - The client ID of the AAD registered application.

client_secret

Required - The client secret of the AAD registered application ().

ingestion_endpoint

Required - The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net

database_name

Required - The database name.

table_name

Required - The table name.

ingestion_mapping_reference

Optional - The name of a that will be used to map the ingested payload into the table columns.

log_key

Key name of the log content.

log

include_tag_key

If enabled, a tag is appended to output. The key name is used tag_key property.

On

tag_key

The key name of tag. If include_tag_key is false, This property is ignored.

tag

include_time_key

If enabled, a timestamp is appended to output. The key name is used time_key property.

On

time_key

The key name of time. If include_time_key is false, This property is ignored.

timestamp

Configuration File

Get started quickly with this configuration file:

[OUTPUT]
    Match *
    Name azure_kusto
    Tenant_Id <app_tenant_id>
    Client_Id <app_client_id>
    Client_Secret <app_secret>
    Ingestion_Endpoint https://ingest-<cluster>.<region>.kusto.windows.net
    Database_Name <database_name>
    Table_Name <table_name>
    Ingestion_Mapping_Reference <mapping_name>

Troubleshooting

403 Forbidden

If you get a 403 Forbidden error response, make sure that:

You provided the correct AAD registered application credentials.
You authorized the application to ingest into your database or table.

Amazon S3

Send logs, data, metrics to Amazon S3

The Amazon S3 output plugin allows you to ingest your records into the S3 cloud object store.

The plugin can upload data to S3 using the multipart upload API or using S3 PutObject. Multipart is the default and is recommended; Fluent Bit will stream data in a series of 'parts'. This limits the amount of data it has to buffer on disk at any point in time. By default, every time 5 MiB of data have been received, a new 'part' will be uploaded. The plugin can create files up to gigabytes in size from many small chunks/parts using the multipart API. All aspects of the upload process are configurable using the configuration options.

The plugin allows you to specify a maximum file size, and a timeout for uploads. A file will be created in S3 when the max size is reached, or the timeout is reached- whichever comes first.

Records are stored in files in S3 as newline delimited JSON.

See here for details on how AWS credentials are fetched.

NOTE: The Prometheus success/retry/error metrics values outputted by Fluent Bit's built-in http server are meaningless for the S3 output. This is because S3 has its own buffering and retry mechanisms. The Fluent Bit AWS S3 maintainers apologize for this feature gap; you can track our progress fixing it on GitHub.

Configuration Parameters

Key

Description

Default

region

The AWS region of your S3 bucket

us-east-1

bucket

S3 Bucket name

None

json_date_key

Specify the name of the time key in the output record. To disable the time key just set the value to false.

date

json_date_format

Specify the format of the date. Supported formats are double, epoch, iso8601 (eg: 2018-05-30T09:39:52.000681Z) and java_sql_timestamp (eg: 2018-05-30 09:39:52.000681)

iso8601

total_file_size

Specifies the size of files in S3. Minimum size is 1M. With use_put_object On the maximum size is 1G. With multipart upload mode, the maximum size is 50G.

100M

upload_chunk_size

The size of each 'part' for multipart uploads. Max: 50M

5,242,880 bytes

upload_timeout

Whenever this amount of time has elapsed, Fluent Bit will complete an upload and create a new file in S3. For example, set this value to 60m and you will get a new file every hour.

10m

store_dir

Directory to locally buffer data before sending. When multipart uploads are used, data will only be buffered until the upload_chunk_size is reached. S3 will also store metadata about in progress multipart uploads in this directory; this allows pending uploads to be completed even if Fluent Bit stops and restarts. It will also store the current $INDEX value if enabled in the S3 key format so that the $INDEX can keep incrementing from its previous value after Fluent Bit restarts.

/tmp/fluent-bit/s3

store_dir_limit_size

The size of the limitation for disk usage in S3. Limit the amount of s3 buffers in the store_dir to limit disk usage. Note: Use store_dir_limit_size instead of storage.total_limit_size which can be used to other plugins, because S3 has its own buffering system.

0, which means unlimited

s3_key_format

Format string for keys in S3. This option supports a UUID, strftime time formatters, a syntax for selecting parts of the Fluent log tag using a syntax inspired by the rewrite_tag filter. Add $UUID in the format string to insert a random string. Add $INDEX in the format string to insert an integer that increments each upload. The $INDEX value will be saved in the store_dir so that if Fluent Bit restarts the value will keep incrementing from the previous run. Add $TAG in the format string to insert the full log tag; add $TAG[0] to insert the first part of the tag in the s3 key. The tag is split into “parts” using the characters specified with the s3_key_format_tag_delimiters option. Add extension directly after the last piece of the format string to insert a key suffix. If you want to specify a key suffix and you are in use_put_object mode, you must specify $UUID as well. More explanations can be found in the S3 Key Format explainer section further down in this document. See the in depth examples and tutorial in the documentation. Time in s3_key is the timestamp of the first record in the S3 file.

/fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S

s3_key_format_tag_delimiters

A series of characters which will be used to split the tag into 'parts' for use with the s3_key_format option. See the in depth examples and tutorial in the documentation.

static_file_path

Disables behavior where UUID string is automatically appended to end of S3 key name when $UUID is not provided in s3_key_format. $UUID, time formatters, $TAG, and other dynamic key formatters all work as expected while this feature is set to true.

false

use_put_object

Use the S3 PutObject API, instead of the multipart upload API. When this option is on, key extension is only available when $UUID is specified in s3_key_format. If $UUID is not included, a random string will be appended at the end of the format string and the key extension cannot be customized in this case.

false

role_arn

ARN of an IAM role to assume (ex. for cross account access).

None

endpoint

Custom endpoint for the S3 API. An endpoint can contain scheme and port.

None

sts_endpoint

Custom endpoint for the STS API.

None

canned_acl

None

compression

Compression type for S3 objects. 'gzip' is currently the only supported value by default. If Apache Arrow support was enabled at compile time, you can also use 'arrow'. For gzip compression, the Content-Encoding HTTP Header will be set to 'gzip'. Gzip compression can be enabled when use_put_object is 'on' or 'off' (PutObject and Multipart). Arrow compression can only be enabled with use_put_object On.

None

content_type

A standard MIME type for the S3 object; this will be set as the Content-Type HTTP header.

None

send_content_md5

Send the Content-MD5 header with PutObject and UploadPart requests, as is required when Object Lock is enabled.

false

auto_retry_requests

Immediately retry failed requests to AWS services once. This option does not affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which may help improve throughput when there are transient/random networking issues.

true

log_key

By default, the whole log record will be sent to S3. If you specify a key name with this option, then only the value of that key will be sent to S3. For example, if you are using Docker, you can specify log_key log and only the log message will be sent to S3.

None

preserve_data_ordering

Normally, when an upload request fails, there is a high chance for the last received chunk to be swapped with a later chunk, resulting in data shuffling. This feature prevents this shuffling by using a queue logic for uploads.

true

storage_class

None

retry_limit

Integer value to set the maximum number of retries allowed. Note: this configuration is released since version 1.9.10 and 2.0.1. For previous version, the number of retries is 5 and is not configurable.

external_id

Specify an external ID for the STS API, can be used with the role_arn parameter if your role requires an external ID.

None

TLS / SSL

To skip TLS verification, set tls.verify as false. For more details about the properties available and general configuration, please refer to the TLS/SSL section.

Permissions

The plugin requires the following AWS IAM permissions:

{
	"Version": "2012-10-17",
	"Statement": [{
		"Effect": "Allow",
		"Action": [
			"s3:PutObject"
		],
		"Resource": "*"
	}]
}

Differences between S3 and other Fluent Bit outputs

The s3 output plugin is special because its use case is to upload files of non-trivial size to an Amazon S3 bucket. This is in contrast to most other outputs which send many requests to upload data in batches of a few Megabytes or less.

When Fluent Bit recieves logs, it stores them in chunks, either in memory or the filesystem depending on your settings. A chunk is usually around 2 MB in size. Fluent Bit sends the chunks in order to each output that matches their tag. Most outputs then send the chunk immediately to their destination. A chunk is sent to the output's "flush callback function", which must return one of FLB_OK, FLB_RETRY, or FLB_ERROR. Fluent Bit keeps count of the return values from each outputs "flush callback function"; these counters are the data source for Fluent Bit's error, retry, and success metrics available in prometheus format via its monitoring interface.

The S3 output plugin is a Fluent Bit output plugin and thus it conforms to the Fluent Bit output plugin specification. However, since the S3 use case is to upload large files, generally much larger than 2 MB, its behavior is different. The S3 "flush callback function" simply buffers the incoming chunk to the filesystem, and returns an FLB_OK. Consequently, the prometheus metrics available via the Fluent Bit http server are meaningless for S3. In addition, the storage.total_limit_size parameter is not meaningful for S3 since it has its own buffering system in the store_dir. Instead, use store_dir_limit_size. Finally, S3 always requires a write-able filesystem; running Fluent Bit on a read-only filesystem will not work with the S3 output.

S3 uploads are primarily initiated via the S3 "timer callback function", which runs separately from its "flush callback function". Because S3 has its own system of buffering and its own callback to upload data, the normal sequential data ordering of chunks provided by the Fluent Bit engine may be compromised. Consequently, S3 has the presevere_data_ordering option which will ensure data is uploaded in the original order it was collected by Fluent Bit.

Summary: Uniqueness in S3 Plugin

The HTTP Monitoring interface output metrics are not meaningful for S3: AWS understands that this is non-ideal; we have opened an issue with a design that will allow S3 to manage its own output metrics.
You must use store_dir_limit_size to limit the space on disk used by S3 buffer files.
The original ordering of data inputted to Fluent Bit may not be preserved unless you enable preserve_data_ordering On.

S3 Key Format and Tag Delimiters

In Fluent Bit, all logs have an associated tag. The