Comment on page

Azure Logs Ingestion API

Send logs to Azure Log Analytics using Logs Ingestion API with DCE and DCR
Azure Logs Ingestion plugin allows you ingest your records using Logs Ingestion API in Azure Monitor to supported Azure tables or to custom tables that you create.
The Logs ingestion API requires the following components:
  • A Data Collection Endpoint (DCE)
  • A Data Collection Rule (DCR) and
  • A Log Analytics Workspace
Note: According to this document, all resources should be in the same region.
To visualize basic Logs Ingestion operation, see the following image:
To get more details about how to setup these components, please refer to the following documentations:

Configuration Parameters

Required - The tenant ID of the AAD application.
Required - The client ID of the AAD application.
Required - The client secret of the AAD application (App Secret).
Required - Data Collection Endpoint(DCE) URL.
Required - Data Collection Rule (DCR) immutable ID (see this document to collect the immutable id)
Required - The name of the custom log table (include the _CL suffix as well if applicable)
Optional - Specify the key name where the timestamp will be stored.
Optional - If enabled, will generate a timestamp and append it to JSON. The key name is set by the 'time_key' parameter.
Optional - Enable HTTP payload gzip compression.

Getting Started

To send records into an Azure Log Analytics using Logs Ingestion API the following resources needs to be created:
  • A Data Collection Endpoint (DCE) for ingestion
  • A Data Collection Rule (DCR) for data transformation
  • Either an Azure tables or custom tables
  • An app registration with client secrets (for DCR access).
You can follow this guideline to setup the DCE, DCR, app registration and a custom table.

Configuration File

Use this configuration to quickly get started:
Name tail
Path /path/to/your/sample.log
Tag sample
Key RawData
# Or use other plugins Plugin
# Name cpu
# Tag sample
Name modify
Match sample
# Add a json key named "Application":"fb_log"
Add Application fb_log
# Enable this section to see your json-log format
# Name stdout
# Match *
Name azure_logs_ingestion
Match sample
client_id XXXXXXXX-xxxx-yyyy-zzzz-xxxxyyyyzzzzxyzz
client_secret some.secret.xxxzzz
tenant_id XXXXXXXX-xxxx-yyyy-zzzz-xxxxyyyyzzzzxyzz
dcr_id dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
table_name ladcr_CL
time_generated true
time_key Time
Compress true
Setup your DCR transformation accordingly based on the json output from fluent-bit's pipeline (input, parser, filter, output).
Last modified 2mo ago