The ECS Filter Enriches logs with AWS Elastic Container Service Metadata. The plugin can enrich logs with task, cluster and container metadata. The plugin uses the ECS Agent introspection API to obtain metadata. This filter only works with the ECS EC2 launch type. The filter only works when Fluent Bit is running on an ECS EC2 Container Instance and has access to the ECS Agent introspection API. The filter is not supported on ECS Fargate. To obtain metadata on ECS Fargate, use the built-in FireLens metadata or the AWS for Fluent Bit init project.
The plugin supports the following configuration parameters:
| Key | Description | Default |
|---|---|---|
The following template variables can be used for values with the Add option. See the tutorial below for examples.
| Variable | Description | Supported with Cluster_Metadata_Only On |
|---|---|---|
The output log should be similar to:
The output log would be similar to:
Notice that the template variables in the value for the resource key are separated by dot characters. Please see the section below about limitations in which characters can be used to separate template variables.
This examples shows a use case for the Cluster_Metadata_Only option- attaching cluster metadata to ECS Agent logs.
Notice in example 2, that the template values are separated by dot characters. This is important; the Fluent Bit record_accessor library has a limitation in the characters that can separate template variables- only dots and commas (. and ,) can come after a template variable. This is because the templating library must parse the template and determine the end of a variable.
The following would be invalid templates because the two template variables are not separated by commas or dots:
$TaskID-$ECSContainerName
$TaskID/$ECSContainerName
$TaskID_$ECSContainerName
$TaskIDfooo$ECSContainerName
However, the following are valid:
$TaskID.$ECSContainerName
$TaskID.ecs_resource.$ECSContainerName
$TaskID.fooo.$ECSContainerName
And the following are valid since they only contain one template variable with nothing after it:
fooo$TaskID
fooo____$TaskID
fooo/bar$TaskID
Add
This parameter is similar to the ADD option in the modify filter. You can specify it any number of times and it takes two arguments, a KEY name and VALUE. The value uses Fluent Bit record_accessor syntax to create a template that uses ECS Metadata values. See the list below for supported metadata templating keys. This option is designed to give you full power to control both the key names for metadata as well as the format for metadata values. See the examples below for more.
No default
ECS_Tag_Prefix
This parameter is similar to the Kube_Tag_Prefix option in the Kubernetes filter and performs the same function. The full log tag should be prefixed with this string and after the prefix the filter must find the next characters in the tag to be the Docker Container Short ID (the first 12 characters of the full container ID). The filter uses this to identify which container the log came from so it can find which task it is a part of. See the design section below for more information. If not specified, it defaults to empty string, meaning that the tag must be prefixed with the 12 character container short ID. If you just want to attach cluster metadata to system/OS logs from processes that do not run as part of containers or ECS Tasks, then do not set this parameter and enable the Cluster_Metadata_Only option
emptry string
Cluster_Metadata_Only
When enabled, the plugin will only attempt to attach cluster metadata values. This is useful if you want to attach cluster metadata to system/OS logs from processes that do not run as part of containers or ECS Tasks.
Off
ECS_Meta_Cache_TTL
The filter builds a hash table in memory mapping each unique container short ID to its metadata. This option sets a max TTL for objects in the hash table. You should set this if you have frequent container/task restarts. For example, your cluster runs short running batch jobs that complete in less than 10 minutes, there is no reason to keep any stored metadata longer than 10 minutes. So you would set this parameter to "10m".
1h (1 hour)
$ClusterName
The ECS cluster name. Fluent Bit is running on EC2 instance(s) that are part of this cluster.
Yes
$ContainerInstanceARN
The full ARN of the ECS EC2 Container Instance. This is the instance that Fluent Bit is running on.
Yes
$ContainerInstanceID
The ID of the ECS EC2 Container Instance.
Yes
$ECSAgentVersion
The Version string of the ECS Agent that is running on the container instance.
Yes
$ECSContainerName
The name of the container from which the log originated. This is the name in your ECS Task Definition.
No
$DockerContainerName
The name of the container from which the log originated. This is the name obtained from Docker and is the name shown if you run docker ps on the instance.
No
$ContainerID
The ID of the container from which the log originated. This is the full 64 character long container ID.
No
$TaskDefinitionFamily
The family name of the task definition for the task from which the log originated.
No
$TaskDefinitionVersion
The version/revision of the task definition for the task from which the log originated.
No
$TaskID
The ID of the ECS Task from which the log originated.
No
$TaskARN
The full ARN of the ECS Task from which the log originated.
No