Azure Blob
Microsoft certified Azure Storage Blob connector
The Azure Blob output plugin allows ingesting your records into Azure Blob Storage service. This connector is designed to use the Append Blob and Block Blob API.
The Fluent Bit plugin works with the official Azure Service and can be configured to be used with a service emulator such as Azurite.
Azure Storage account
Ensure you have an Azure Storage account. Azure Blob Storage Tutorial (video) explains how to set up your account.
Configuration parameters
Fluent Bit exposes the following configuration properties.
account_name
Azure Storage account name.
none
auth_type
Specify the type to authenticate against the service. Supported values: key, sas.
key
auto_create_container
If container_name doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container.
true
azure_blob_buffer_key
Set the Azure Blob buffer key which needs to be specified when using multiple instances of Azure Blob output plugin and buffering is enabled.
key
blob_type
Specify the desired blob type. Supported values: appendblob, blockblob.
appendblob
blob_uri_length
Set the length of the generated blob URI used when creating and uploading objects to Azure Blob Storage.
64
buffer_dir
Specifies the location of directory where the buffered data will be stored.
/tmp/fluent-bit/azure-blob/
buffer_file_delete_early
Whether to delete the buffered file early after successful blob creation.
false
buffering_enabled
Enable buffering into disk before ingesting into Azure Blob.
false
compress
Sets payload compression in network transfer. Supported values: gzip, zstd.
none
compress_blob
Enables compression in the final blockblob file. When enabled without compress, it uses GZIP; if compress is also set, it inherits that codec. This option isn't compatible when blob_type = appendblob. Fluent Bit returns a configuration error and fails to start.
false
configuration_endpoint_bearer_token
Bearer token for the configuration endpoint.
none
configuration_endpoint_password
Basic authentication password for the configuration endpoint.
none
configuration_endpoint_url
Configuration endpoint URL.
none
configuration_endpoint_username
Basic authentication username for the configuration endpoint.
none
container_name
Name of the container that will contain the blobs.
none
database_file
Absolute path to a database file used to store blob file contexts.
none
date_key
Key name used to store the record timestamp.
@timestamp
delete_on_max_upload_error
Whether to delete the buffer file on maximum upload errors.
false
emulator_mode
To send data to an Azure emulator service like Azurite, enable this option to format the requests in the expected format.
false
endpoint
When using an emulator, this option lets you specify the absolute HTTP address of such service. For example, http://127.0.0.1:10000.
none
file_delivery_attempt_limit
Maximum number of delivery attempts for a file.
1
io_timeout
HTTP IO timeout.
60s
part_delivery_attempt_limit
Maximum number of delivery attempts for a file part.
1
part_size
Size of each part when uploading blob files.
25M
path
Optional. The path to store your blobs. If your blob name is myblob, specify subdirectories for storage using path. For example, setting path to /logs/kubernetes will store your blob in /logs/kubernetes/myblob.
none
sas_token
Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when auth_type is sas.
none
scheduler_max_retries
Maximum number of retries for the scheduler send blob.
3
shared_key
Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when auth_type is key.
none
store_dir_limit_size
Set the max size of the buffer directory.
8G
tls
Enable or disable TLS encryption. Azure service requires this to be set to on.
off
unify_tag
Whether to create a single buffer file when buffering mode is enabled.
false
upload_file_size
Specifies the size of files to be uploaded in MB.
200M
upload_part_freshness_limit
Maximum lifespan of an uncommitted file part.
6D
upload_parts_timeout
Timeout for uploading parts of a blob file.
10M
upload_timeout
Optional. Specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than x minutes and haven't reached upload_file_size limit yet.
30m
Get started
Fluent Bit can deliver records to the official service or an emulator.
Configuration for Azure Storage
The following configuration example generates a random message with a custom tag:
After you run the configuration file, you will be able to query the data using the Azure Storage Explorer. The example generates the following content in the explorer:
Configuring and using Azure Emulator: Azurite
Install and run Azurite
Install Azurite using
npm:Run the service:
The command should return results similar to:
Configuring Fluent Bit for Azurite
Azurite comes with a default account_name and shared_key. Instead of the defaults, be sure to use the specific values provided in the following example:
After running the Fluent Bit configuration, you will see the data flowing into Azurite:
Last updated
Was this helpful?