Azure Data Explorer

Send logs to Azure Data Explorer (Kusto)

The Kusto output plugin allows to ingest your logs into an Azure Data Explorer cluster, via the Queued Ingestion mechanism. This output plugin can also be used to ingest logs into an Eventhouse cluster in Microsoft Fabric Real Time Analytics.

For ingesting into Azure Data Explorer: Creating a Kusto Cluster and Database

You can create an Azure Data Explorer cluster in one of the following ways:

• Create a free-tier cluster

• Create a fully featured cluster

For ingesting into Microsoft Fabric Real Time Analytics : Creating an Eventhouse Cluster and KQL Database

You can create an Eventhouse cluster and a KQL database follow the following steps:

• Create an Eventhouse cluster

• Create a KQL database

Creating an Azure Registered Application

Fluent-Bit will use the application's credentials, to ingest data into your cluster.

• Register an Application

• Add a client secret

• Authorize the app in your database

Creating a Table

Fluent-Bit ingests the event data into Kusto in a JSON format, that by default will include 3 properties:

• log - the actual event payload.

• tag - the event tag.

• timestamp - the event timestamp.

A table with the expected schema must exist in order for data to be ingested properly.

.create table FluentBit (log:dynamic, tag:string, timestamp:datetime)

Optional - Creating an Ingestion Mapping

By default, Kusto will insert incoming ingestions into a table by inferring the mapped table columns, from the payload properties. However, this mapping can be customized by creatng a JSON ingestion mapping. The plugin can be configured to use an ingestion mapping via the ingestion_mapping_reference configuration key.

Configuration Parameters

Key	Description	Default
tenant_id	Required if managed_identity_client_id is not set - The tenant/domain ID of the AAD registered application.
client_id	Required if managed_identity_client_id is not set - The client ID of the AAD registered application.
client_secret	Required if managed_identity_client_id is not set - The client secret of the AAD registered application (App Secret).
managed_identity_client_id	Required if tenant_id, client_id, and client_secret are not set - The managed identity ID to authenticate with. Set to SYSTEM for system-assigned managed identity, or set to the MI client ID (GUID) for user-assigned managed identity.
ingestion_endpoint	Required - The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net
database_name	Required - The database name.
table_name	Required - The table name.
ingestion_mapping_reference	Optional - The name of a JSON ingestion mapping that will be used to map the ingested payload into the table columns.
log_key	Key name of the log content.	log
include_tag_key	If enabled, a tag is appended to output. The key name is used tag_key property.	On
tag_key	The key name of tag. If include_tag_key is false, This property is ignored.	tag
include_time_key	If enabled, a timestamp is appended to output. The key name is used time_key property.	On
time_key	The key name of time. If include_time_key is false, This property is ignored.	timestamp
ingestion_endpoint_connect_timeout	The connection timeout of various Kusto endpoints in seconds.	60
compression_enabled	If enabled, sends compressed HTTP payload (gzip) to Kusto.	true
ingestion_resources_refresh_interval	The ingestion resources refresh interval of Kusto endpoint in seconds.
workers	The number of workers to perform flush operations for this output.	0
buffering_enabled	Optional - Enable buffering into disk before ingesting into Azure Kusto.	Off
buffer_dir	Optional - When buffering is On, specifies the location of directory where the buffered data will be stored.	/tmp/fluent-bit/azure-kusto/
upload_timeout	Optional - When buffering is On, specifies a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than x minutes and haven't reached upload_file_size limit.	30m
upload_file_size	Optional - When buffering is On, specifies the size of files to be uploaded in MBs.	200MB
azure_kusto_buffer_key	Optional - When buffering is On, set the Azure Kusto buffer key which must be specified when using multiple instances of Azure Kusto output plugin and buffering is enabled.	key
store_dir_limit_size	Optional - When buffering is On, set the max size of the buffer directory.	8GB
buffer_file_delete_early	Optional - When buffering is On, whether to delete the buffered file early after successful blob creation.	Off
unify_tag	Optional - This creates a single buffer file when the buffering mode is On.	On
blob_uri_length	Optional - Set the length of generated blob URI before ingesting to Kusto.	64
scheduler_max_retries	Optional - When buffering is On, set the maximum number of retries for ingestion using the scheduler.	3
delete_on_max_upload_error	Optional - When buffering is On, whether to delete the buffer file on maximum upload errors.	Off
io_timeout	Optional - Configure the HTTP IO timeout for uploads.	60s

Configuration File

Get started quickly with this configuration file:

[OUTPUT]
    Match *
    Name azure_kusto
    Tenant_Id <app_tenant_id>
    Client_Id <app_client_id>
    Client_Secret <app_secret>
    Ingestion_Endpoint https://ingest-<cluster>.<region>.kusto.windows.net
    Database_Name <database_name>
    Table_Name <table_name>
    Ingestion_Mapping_Reference <mapping_name>
    ingestion_endpoint_connect_timeout <ingestion_endpoint_connect_timeout>
    compression_enabled <compression_enabled>
    ingestion_resources_refresh_interval <ingestion_resources_refresh_interval>
    buffering_enabled On
    upload_timeout 2m
    upload_file_size 125M
    azure_kusto_buffer_key kusto1
    buffer_file_delete_early Off
    unify_tag On
    buffer_dir /var/log/
    store_dir_limit_size 16GB
    blob_uri_length 128
    scheduler_max_retries 3
    delete_on_max_upload_error Off
    io_timeout 60s

Troubleshooting

403 Forbidden

If you get a 403 Forbidden error response, make sure that:

• You provided the correct AAD registered application credentials.

• You authorized the application to ingest into your database or table.