The Azure Blob output plugin allows ingesting your records into Azure Blob Storage service. This connector is designed to use the Append Blob and Block Blob API.
Our plugin works with the official Azure Service and also can be configured to be used with a service emulator such as Azurite.
Before getting started, make sure you already have an Azure Storage account. As a reference, the following link explains step-by-step how to set up your account:
We expose different configuration properties. The following table lists all the options available, and the next section has specific configuration details for the official service or the emulator.
Key | Description | default |
account_name | Azure Storage account name. This configuration property is mandatory | ​ |
shared_key | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory. | ​ |
container_name | Name of the container that will contain the blobs. This configuration property is mandatory | ​ |
blob_type | Specify the desired blob type. Fluent Bit supports | appendblob |
auto_create_container | If | on |
path | Optional path to store your blobs. If your blob name is | ​ |
emulator_mode | If you want to send data to an Azure emulator service like Azurite, enable this option so the plugin will format the requests to the expected format. | off |
endpoint | If you are using an emulator, this option allows you to specify the absolute HTTP address of such service. e.g: http://127.0.0.1:10000. | ​ |
tls | Enable or disable TLS encryption. Note that Azure service requires this to be turned on. | off |
As mentioned above, you can either deliver records to the official service or an emulator. Below we have an example for each use case.
The following configuration example generates a random message with a custom tag:
[SERVICE]flush 1log_level info​[INPUT]name dummydummy {"name": "Fluent Bit", "year": 2020}samples 1tag var.log.containers.app-default-96cbdef2340.log​[OUTPUT]name azure_blobmatch *account_name YOUR_ACCOUNT_NAMEshared_key YOUR_SHARED_KEYpath kubernetescontainer_name logsauto_create_container ontls on
After you run the configuration file above, you will be able to query the data using the Azure Storage Explorer. The example above will generate the following content in the explorer:
The quickest way to get started is to install Azurite using npm:
$ npm install -g azurite
then run the service:
$ azuriteAzurite Blob service is starting at http://127.0.0.1:10000Azurite Blob service is successfully listening at http://127.0.0.1:10000Azurite Queue service is starting at http://127.0.0.1:10001Azurite Queue service is successfully listening at http://127.0.0.1:10001
​Azurite comes with a default account_name
and shared_key
, so make sure to use the specific values provided in the example below (do an exact copy/paste):
[SERVICE]flush 1log_level info​[INPUT]name dummydummy {"name": "Fluent Bit", "year": 2020}samples 1tag var.log.containers.app-default-96cbdef2340.log​[OUTPUT]name azure_blobmatch *account_name devstoreaccount1shared_key Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==path kubernetescontainer_name logsauto_create_container ontls offemulator_mode onendpoint http://127.0.0.1:10000
after running that Fluent Bit configuration you will see the data flowing into Azurite:
$ azuriteAzurite Blob service is starting at http://127.0.0.1:10000Azurite Blob service is successfully listening at http://127.0.0.1:10000Azurite Queue service is starting at http://127.0.0.1:10001Azurite Queue service is successfully listening at http://127.0.0.1:10001127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "GET /devstoreaccount1/logs?restype=container HTTP/1.1" 404 -127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs?restype=container HTTP/1.1" 201 -127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log?comp=appendblock HTTP/1.1" 404 -127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log HTTP/1.1" 201 -127.0.0.1 - - [03/Sep/2020:17:40:04 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log?comp=appendblock HTTP/1.1" 201 -