Links

ECS Metadata

The ECS Filter Enriches logs with AWS Elastic Container Service Metadata. The plugin can enrich logs with task, cluster and container metadata. The plugin uses the ECS Agent introspection API to obtain metadata. This filter only works with the ECS EC2 launch type. The filter only works when Fluent Bit is running on an ECS EC2 Container Instance and has access to the ECS Agent introspection API. The filter is not supported on ECS Fargate. To obtain metadata on ECS Fargate, use the built-in FireLens metadata or the AWS for Fluent Bit init project.

Configuration Parameters

The plugin supports the following configuration parameters:
Key
Description
Default
Add
This parameter is similar to the ADD option in the modify filter. You can specify it any number of times and it takes two arguments, a KEY name and VALUE. The value uses Fluent Bit record_accessor syntax to create a template that uses ECS Metadata values. See the list below for supported metadata templating keys. This option is designed to give you full power to control both the key names for metadata as well as the format for metadata values. See the examples below for more.
No default
ECS_Tag_Prefix
This parameter is similar to the Kube_Tag_Prefix option in the Kubernetes filter and performs the same function. The full log tag should be prefixed with this string and after the prefix the filter must find the next characters in the tag to be the Docker Container Short ID (the first 12 characters of the full container ID). The filter uses this to identify which container the log came from so it can find which task it is a part of. See the design section below for more information. If not specified, it defaults to empty string, meaning that the tag must be prefixed with the 12 character container short ID. If you just want to attach cluster metadata to system/OS logs from processes that do not run as part of containers or ECS Tasks, then do not set this parameter and enable the Cluster_Metadata_Only option
emptry string
Cluster_Metadata_Only
When enabled, the plugin will only attempt to attach cluster metadata values. This is useful if you want to attach cluster metadata to system/OS logs from processes that do not run as part of containers or ECS Tasks.
Off
ECS_Meta_Cache_TTL
The filter builds a hash table in memory mapping each unique container short ID to its metadata. This option sets a max TTL for objects in the hash table. You should set this if you have frequent container/task restarts. For example, your cluster runs short running batch jobs that complete in less than 10 minutes, there is no reason to keep any stored metadata longer than 10 minutes. So you would set this parameter to "10m".
1h (1 hour)

Supported Templating Variables for the ADD option

The following template variables can be used for values with the Add option. See the tutorial below for examples.
Variable
Description
Supported with Cluster_Metadata_Only On
$ClusterName
The ECS cluster name. Fluent Bit is running on EC2 instance(s) that are part of this cluster.
Yes
$ContainerInstanceARN
The full ARN of the ECS EC2 Container Instance. This is the instance that Fluent Bit is running on.
Yes
$ContainerInstanceID
The ID of the ECS EC2 Container Instance.
Yes
$ECSAgentVersion
The Version string of the ECS Agent that is running on the container instance.
Yes
$ECSContainerName
The name of the container from which the log originated. This is the name in your ECS Task Definition.
No
$DockerContainerName
The name of the container from which the log originated. This is the name obtained from Docker and is the name shown if you run docker ps on the instance.
No
$ContainerID
The ID of the container from which the log originated. This is the full 64 character long container ID.
No
$TaskDefinitionFamily
The family name of the task definition for the task from which the log originated.
No
$TaskDefinitionVersion
The version/revision of the task definition for the task from which the log originated.
No
$TaskID
The ID of the ECS Task from which the log originated.
No
$TaskARN
The full ARN of the ECS Task from which the log originated.
No

Configuration File

Example 1: Attach Task ID and cluster name to container logs

[INPUT]
Name tail
Tag ecs.*
Path /var/lib/docker/containers/*/*.log
Docker_Mode On
Docker_Mode_Flush 5
Docker_Mode_Parser container_firstline
Parser docker
DB /var/fluent-bit/state/flb_container.db
Mem_Buf_Limit 50MB
Skip_Long_Lines On
Refresh_Interval 10
Rotate_Wait 30
storage.type filesystem
Read_From_Head Off
[FILTER]
Name ecs
Match *
ECS_Tag_Prefix ecs.var.lib.docker.containers.
ADD ecs_task_id $TaskID
ADD cluster $ClusterName
[OUTPUT]
Name stdout
Match *
Format json_lines
The output log should be similar to:
{
"date":1665003546.0,
"log":"some message from your container",
"ecs_task_id" "1234567890abcdefghijklmnop",
"cluster": "your_cluster_name",
}

Example 2: Attach customized resource name to container logs

[INPUT]
Name tail
Tag ecs.*
Path /var/lib/docker/containers/*/*.log
Docker_Mode On
Docker_Mode_Flush 5
Docker_Mode_Parser container_firstline
Parser docker
DB /var/fluent-bit/state/flb_container.db
Mem_Buf_Limit 50MB
Skip_Long_Lines On
Refresh_Interval 10
Rotate_Wait 30
storage.type filesystem
Read_From_Head Off
[FILTER]
Name ecs
Match *
ECS_Tag_Prefix ecs.var.lib.docker.containers.
ADD resource $ClusterName.$TaskDefinitionFamily.$TaskID.$ECSContainerName
[OUTPUT]
Name stdout
Match *
Format json_lines
The output log would be similar to:
{
"date":1665003546.0,
"log":"some message from your container",
"resource" "cluster.family.1234567890abcdefghijklmnop.app",
}
Notice that the template variables in the value for the resource key are separated by dot characters. Please see the section below about limitations in which characters can be used to separate template variables.

Example 3: Attach cluster metadata to non-container logs

This examples shows a use case for the Cluster_Metadata_Only option- attaching cluster metadata to ECS Agent logs.
[INPUT]
Name tail
Tag ecsagent.*
Path /var/log/ecs/*
DB /var/fluent-bit/state/flb_ecs.db
Mem_Buf_Limit 50MB
Skip_Long_Lines On
Refresh_Interval 10
Rotate_Wait 30
storage.type filesystem
# Collect all logs on instance
Read_From_Head On
[FILTER]
Name ecs
Match *
Cluster_Metadata_Only On
ADD cluster $ClusterName
[OUTPUT]
Name stdout
Match *
Format json_lines

Limitations of record_accessor templating

Notice in example 2, that the template values are separate by dot characters. This is important; the Fluent Bit record_accessor library has a limitation in the characters that can separate template variables- only dots and commas (. and ,) can come after a template variable. This is because the templating library must parse the template and determine the end of a variable.
The following would be invalid templates because the two template variables are not separated by commas or dots:
  • $TaskID-$ECSContainerName
  • $TaskID/$ECSContainerName
  • $TaskID_$ECSContainerName
  • $TaskIDfooo$ECSContainerName
However, the following are valid:
  • $TaskID.$ECSContainerName
  • $TaskID.ecs_resource.$ECSContainerName
  • $TaskID.fooo.$ECSContainerName
And the following are valid since they only contain one template variable with nothing after it:
  • fooo$TaskID
  • fooo____$TaskID
  • fooo/bar$TaskID
Last modified 1mo ago