Kubernetes
Fluent Bit Kubernetes filter enriches your log files with Kubernetes metadata.
When Fluent Bit is deployed in Kubernetes as a DaemonSet and configured to read the log files from the containers (using tail
or systemd
input plugins), this filter can perform the following operations:
Analyze the Tag and extract the following metadata:
Pod Name
Namespace
Container Name
Container ID
Query Kubernetes API Server or Kubelet to obtain extra metadata for the pod in question:
Pod ID
Labels
Owner References
Annotations
Namespace Labels
Namespace Annotations
The data is cached locally in memory and appended to each record.
Configuration parameters
The plugin supports the following configuration parameters:
Buffer_Size
32k
Kube_URL
API Server end-point
Kube_CA_File
CA certificate file
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_CA_Path
Absolute path to scan for certificate files
none
Kube_Token_File
Token file
/var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix
When the source records come from the tail
input plugin, this option specifies the prefix used in tail
configuration.
kube.var.log.containers.
Merge_Log
When enabled, check if the log
field content is a JSON string map. If it is, append the map fields as part of the log structure.
Off
Merge_Log_Key
When Merge_Log
is enabled, the filter assumes the log
field from the incoming message is a JSON string message and attempts to create a structured representation of it at the same level of the log
field in the map. If Merge_Log_Key
is set (a string name), all the new structured fields taken from the original log
content are inserted under the new key.
none
Merge_Log_Trim
When Merge_Log
is enabled, trim (remove possible or \r\
) field values.
On
Merge_Parser
Optional parser name to specify how to parse the data contained in the log
key. Recommended for developers or testing only.
none
Keep_Log
When Keep_Log
is disabled and Merge_Log
enabled, the log
field is removed from the incoming message once it has been successfully merged.
On
tls.debug
Debug level between 0
(no information) and 4
(all details).
-1
tls.verify
When enabled, turns on certificate validation when connecting to the Kubernetes API server.
On
tls.verify_hostname
When enabled, turns on hostname validation for certificates.
Off
Use_Journal
When enabled, the filter reads logs in Journald
format.
Off
Cache_Use_Docker_Id
When enabled, metadata will be fetched from Kubernetes when docker_id
is changed.
Off
Regex_Parser
none
K8S-Logging.Parser
Allow Kubernetes pods to suggest a pre-defined parser.
Off
K8S-Logging.Exclude
Allow Kubernetes pods to exclude their logs from the log processor.
Off
Labels
Include Kubernetes pod resource labels in the extra metadata.
On
Annotations
Include Kubernetes pod resource annotations in the extra metadata.
On
Kube_meta_preload_cache_dir
If set, Kubernetes metadata can be cached or pre-loaded from files in JSON format in this directory, named namespace-pod.meta
.
none
Dummy_Meta
If set, use dummy-meta data (for test/dev purposes).
Off
DNS_Retries
Number of DNS lookup retries until the network starts working.
6
DNS_Wait_Time
DNS lookup interval between network status checks.
30
Use_Kubelet
Off
Use_Tag_For_Meta
Off
Kubelet_Port
Kubelet port to use for HTTP requests. This only works when Use_Kubelet
is set to On
.
10250
Kubelet_Host
Kubelet host to use for HTTP requests. This only works when Use_Kubelet
is set to On
.
127.0.0.1
Kube_Meta_Cache_TTL
Configurable time-to-live for Kubernetes cached pod metadata. By default, it's set to 0
which means TTL
for cache entries is disabled and cache entries are evicted at random when capacity is reached. To enable this option, set the number to a time interval. For example, set the value to 60
or 60s
and cache entries which have been created more than 60 seconds ago will be evicted.
0
Kube_Token_TTL
Configurable time-to-live for the Kubernetes token. After this time, the token is reloaded from Kube_Token_File
or the Kube_Token_Command
.
600
Kube_Token_Command
Command to get Kubernetes authorization token. Defaults to NULL
uses the token file to get the token. To manually choose a command to get it, set the command here. For example, run aws-iam-authenticator -i your-cluster-name token --token-only
to set token. This option is currently Linux-only.
NULL
Kube_Meta_Namespace_Cache_TTL
Configurable time-to-live for Kubernetes cached namespace metadata. If set to 0
, entries are evicted at random when capacity is reached.
900
(seconds)
Namespace_Labels
Off
Namespace_Annotations
Off
Namespace_Metadata_Only
Off
Owner_References
Include Kubernetes owner references in the extra metadata.
Off
Processing the log
value
log
valueKubernetes filter provides several ways to process the data contained in the log
key. The following explanation of the workflow assumes that your original Docker parser defined in parsers.conf
is as follows:
To avoid data-type conflicts in Fluent Bit v1.2 or greater, don't use decoders (Decode_Field_As
) if you're using Elasticsearch database in the output.
To perform processing of the log
key, you must enable the Merge_Log
configuration property in this filter, then the following processing order will be done:
If a pod suggests a parser, the filter will use that parser to process the content of
log
.If the
Merge_Parser
option was set and the pod didn't suggest a parser, process thelog
content using the suggested parser in the configuration.If no pod was suggested and
Merge_Parser
isn't set, try to handle the content as JSON.
If log
value processing fails, the value is untouched. The order of processing isn't chained, meaning it's exclusive and the filter will try only one of the options, not all of them.
Kubernetes namespace meta
Enable namespace meta using the following settings:
Namespace_Labels
Namespace_Annotations
Using any namespace meta requires the use of the Kube API. It can't be fetched directly from Kubelet. If Use_Kubelet On
has been set, the Kubelet API will be used only to fetch pod metadata, while namespace meta is fetched from the upstream Kubernetes API.
If collected, namespace meta will be stored in a kubernetes_namespace
record key.
Namespace meta isn't guaranteed to be in sync since namespace labels and annotations can be adjusted after pod creation. Adjust Kube_Meta_Namespace_Cache_TTL
to reduce caching times to fit your use case.
Namespace_Metadata_Only
Using this feature will instruct Fluent Bit to only fetch namespace metadata and to not fetch pod metadata at all. Pod basic metadata like
container id
andhost
won't be added, and the labels and annotations configuration options which are used specifically for pod Metadata will be ignored.
Kubernetes pod annotations
Fluent Bit Kubernetes filters allow Kubernetes pods to suggest certain behaviors for the log processor pipeline when processing the records. It can:
Suggest a pre-defined parser
Request to exclude logs
The following annotations are available:
fluentbit.io/parser[_stream][-container]
Suggest a pre-defined parser. The parser must be registered already by Fluent Bit. This option will only be processed if Fluent Bit configuration (Kubernetes Filter) has enabled the option K8S-Logging.Parser
. If present, the stream (stdout or stderr) will restrict that specific stream. If present, the container can override a specific container in a Pod.
none
fluentbit.io/exclude[_stream][-container]
Define whether to request that Fluent Bit excludes the logs generated by the pod. This option will be processed only if the Fluent Bit configuration (Kubernetes Filter) has enabled the option K8S-Logging.Exclude
.
False
Annotation examples in pod definition
Suggest a parser
The following pod definition runs a pod that emits Apache logs to the standard output. The Annotations suggest that the data should be processed using the pre-defined parser called apache
:
Request to exclude logs
The following example defines a request that the log processor skip the logs from the pod in question:
The annotation value is Boolean which can take a "true"
or "false"
. Values must be quoted.
Kubernetes owner references
An optional feature of Fluent Bit Kubernetes filter includes owner references information under kubernetes.ownerReferences
field in the record when enabled.
For example:
Workflow of Tail and Kubernetes filter
Tail supports tags expansion. If a tag has a star character (*
), it will replace the value with the absolute path of the monitored file, so if your filename and path is:
then the tag for every record of that file becomes:
Slashes (/
) are replaced with dots (.
).
When Kubernetes filter runs, it tries to match all records that start with kube.
. Records from the previous file match the rule and the filter will try to enrich the records.
Kubernetes filter doesn't care from where the logs comes from, but it cares about the absolute name of the monitored file. That information contains the pod name and namespace name that are used to retrieve associated metadata to the running pod from the Kubernetes Master/API Server.
If you have large pod specifications, which can be caused by large numbers of environment variables, increase the Buffer_Size
parameter of the Kubernetes filter. If object sizes exceed this buffer, some metadata will fail to be injected to the logs.
If the configuration property Kube_Tag_Prefix
was configured (available on Fluent Bit >= 1.1.x), it will use that value to remove the prefix that was appended to the Tag in the previous Input
section. The configuration property defaults to kube.var.logs.containers.
, so the previous tag content will be transformed from:
to:
Rather than modify the original tag, the transformation creates a new representation for the filter to perform metadata lookup.
Suggested change The new value is used by the filter to lookup the pod name and namespace, for that purpose it uses an internal regular expression: The new value is used by the filter to lookup the pod name and namespace. For that purpose, it uses an internal regular expression:
Custom regular expressions
Under some uncommon conditions, a user might want to alter that hard-coded regular expression. Use the Regex_Parser
option.
Custom tags For enhanced filtering
One such use case involves splitting logs by namespace, pods, containers or container ID. The tag is restructured within the tail input using match groups. Restructuring can simplify the filtering by those match groups later in the pipeline. Since the tag no longer follows the original filename, a custom Regex_Parser
that matches the new tag structure is required:
The filter can now gather the values of pod_name
and namespace
. With that information, it will check in the local cache (internal hash table) if some metadata for that key pair exists. If it exists, it will enrich the record with the metadata value. Otherwise, it connects to the Kubernetes Master/API Server and retrieves that information.
Using Kubelet to get metadata
Configuration setup
There are some configuration setup needed for this feature.
The following example demonstrates role configuration for the Fluent Bit DaemonSet:
Kubelet needs special permission for the resource nodes/proxy
to get HTTP requests. When creating the role
or clusterRole
, you need to add nodes/proxy
into the rule for resource.
For Fluent Bit configuration, you must set the Use_Kubelet
to true
to enable this feature.
Fluent Bit configuration example:
DaemonSet configuration example:
Set hostNetwork
to true
and dnsPolicy
to ClusterFirstWithHostNet
so the Fluent Bit DaemonSet can call Kubelet locally. Otherwise it can't resolve DNS for kubelet.
Verify the Use_Kubelet
option is working
Use_Kubelet
option is workingTo know if Fluent Bit is using the kubelet, you can review Fluent Bit logs. There should be a log like this:
If you are in debug mode, you can see more:
Troubleshooting
Learn how to solve them to ensure that the Fluent Bit Kubernetes filter is operating properly. The following section describes specific log messages you might receive.
You can't see metadata appended to your pods or other Kubernetes objects
If you aren't seeing metadata added to your Kubernetes logs and see the following in your log message, then you might be facing connectivity issues with the Kubernetes API server.
Potential fix 1: Check Kubernetes roles
When Fluent Bit is deployed as a DaemonSet it generally runs with specific roles that allow the application to talk to the Kubernetes API server. If you are deployed in a more restricted environment ensure that all the Kubernetes roles are set correctly.
You can test this by running the following command. Replace
fluentbit-system
with the namespace where your Fluent Bit is installed.If set roles are configured correctly, it should respond with
yes
.For instance, using Azure AKS, running the previous command might respond with:
If you have can connect to the API server, but still
could not get meta for POD
- debug logging might give you a message withAzure does not have opinion for this user
. The followingsubject
might need to be included in thefluentbit
ClusterRoleBinding
, appended tosubjects
array:Potential fix 2: Check Kubernetes IPv6
There might be cases where you have IPv6 on in the environment and you need to enable this within Fluent Bit. Under the service tag set the following option
ipv6
toon
.Potential fix 3: Check connectivity to
Kube_URL
By default the
Kube_URL
is set tohttps://kubernetes.default.svc:443
. Ensure that you have connectivity to this endpoint from within the cluster and that there are no special permissions interfering with the connection.
You can't see new objects getting metadata
In some cases, you might see only some objects being appended with metadata while other objects aren't enriched. This can occur when local data is cached and doesn't contain the correct ID for the Kubernetes object that requires enrichment. For most Kubernetes objects the Kubernetes API server is updated, which will then be reflected in Fluent Bit logs. In some cases for
Pod
objects, this refresh to the Kubernetes API server can be skipped, causing metadata to be skipped.
Credit
Last updated
Was this helpful?