Azure Data Explorer

Send logs to Azure Data Explorer (Kusto)

The Kusto output plugin allows to ingest your logs into an Azure Data Explorer cluster, via the Queued Ingestion mechanism.

Creating a Kusto Cluster and Database

You can create an Azure Data Explorer cluster in one of the following ways:

Creating an Azure Registered Application

Fluent-Bit will use the application's credentials, to ingest data into your cluster.

Creating a Table

Fluent-Bit ingests the event data into Kusto in a JSON format, that by default will include 3 properties:

log - the actual event payload.
tag - the event tag.
timestamp - the event timestamp.

A table with the expected schema must exist in order for data to be ingested properly.

.create table FluentBit (log:dynamic, tag:string, timestamp:datetime)

Optional - Creating an Ingestion Mapping

By default, Kusto will insert incoming ingestions into a table by inferring the mapped table columns, from the payload properties. However, this mapping can be customized by creatng a JSON ingestion mapping. The plugin can be configured to use an ingestion mapping via the ingestion_mapping_reference configuration key.

Configuration Parameters

Key Description Default

Key	Description	Default
tenant_id	Required - The tenant/domain ID of the AAD registered application.
client_id	Required - The client ID of the AAD registered application.
client_secret	Required - The client secret of the AAD registered application (App Secret).
ingestion_endpoint	Required - The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net
database_name	Required - The database name.
table_name	Required - The table name.
ingestion_mapping_reference	Optional - The name of a JSON ingestion mapping that will be used to map the ingested payload into the table columns.
log_key	Key name of the log content.	`log`
include_tag_key	If enabled, a tag is appended to output. The key name is used `tag_key` property.	`On`
tag_key	The key name of tag. If `include_tag_key` is false, This property is ignored.	`tag`
include_time_key	If enabled, a timestamp is appended to output. The key name is used `time_key` property.	`On`
time_key	The key name of time. If `include_time_key` is false, This property is ignored.	`timestamp`

tenant_id

Required - The tenant/domain ID of the AAD registered application.

client_id

Required - The client ID of the AAD registered application.

client_secret

Required - The client secret of the AAD registered application (App Secret).

ingestion_endpoint

Required - The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net

database_name

Required - The database name.

table_name

Required - The table name.

ingestion_mapping_reference

Optional - The name of a JSON ingestion mapping that will be used to map the ingested payload into the table columns.

log_key

Key name of the log content.

log

include_tag_key

If enabled, a tag is appended to output. The key name is used tag_key property.

On

tag_key

The key name of tag. If include_tag_key is false, This property is ignored.

tag

include_time_key

If enabled, a timestamp is appended to output. The key name is used time_key property.

On

time_key

The key name of time. If include_time_key is false, This property is ignored.

timestamp

Configuration File

Get started quickly with this configuration file:

[OUTPUT]
    Match *
    Name azure_kusto
    Tenant_Id <app_tenant_id>
    Client_Id <app_client_id>
    Client_Secret <app_secret>
    Ingestion_Endpoint https://ingest-<cluster>.<region>.kusto.windows.net
    Database_Name <database_name>
    Table_Name <table_name>
    Ingestion_Mapping_Reference <mapping_name>

Troubleshooting

403 Forbidden

If you get a 403 Forbidden error response, make sure that:

You provided the correct AAD registered application credentials.
You authorized the application to ingest into your database or table.

Last updated 1 year ago