Kubernetes

​Fluent Bit is a lightweight and extensible Log Processor that comes with full support for Kubernetes:
Process Kubernetes containers logs from the file system or Systemd/Journald.
Enrich logs with Kubernetes Metadata.
Centralize your logs in third party storage services like Elasticsearch, InfluxDB, HTTP, etc.
Concepts
Before getting started it is important to understand how Fluent Bit will be deployed. Kubernetes manages a cluster of nodes, so our log agent tool will need to run on every node to collect logs from every POD, hence Fluent Bit is deployed as a DaemonSet (a POD that runs on every node of the cluster).
When Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich each entry with the following information (metadata):
Pod Name
Pod ID
Container Name
Container ID
Labels
Annotations
To obtain this information, a built-in filter plugin called kubernetes talks to the Kubernetes API Server to retrieve relevant information such as the pod_id, labels and annotations, other fields such as pod_name, container_id and container_name are retrieved locally from the log file names. All of this is handled automatically, no intervention is required from a configuration aspect.
Our Kubernetes Filter plugin is fully inspired by the Fluentd Kubernetes Metadata Filter written by Jimmi Dyson.
Installation
​Fluent Bit must be deployed as a DaemonSet, so on that way it will be available on every node of your Kubernetes cluster. To get started run the following commands to create the namespace, service account and role setup:
For Kubernetes v1.21 and below
1
$ kubectl create namespace logging
2
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml
3
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml
4
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml
Copied!
For Kubernetes v1.22
1
$ kubectl create namespace logging
2
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml
3
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-1.22.yaml
4
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding-1.22.yaml
Copied!
The next step is to create a ConfigMap that will be used by our Fluent Bit DaemonSet:
1
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml
Copied!
The default configmap assumes that dockershim is utilized for the cluster. If a CRI runtime, such as containerd or CRI-O, is being utilized, the CRI parser should be utilized. More specifically, change the Parser described in input-kubernetes.conf from docker to cri.
Note for OpenShift
If you are using Red Hat OpenShift you will also need to run the following
1
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-openshift-security-context-constraints.yaml
Copied!
Note for Kubernetes < v1.16
For Kubernetes versions older than v1.16, the DaemonSet resource is not available on apps/v1 , the resource is available on apiVersion: extensions/v1beta1 . Our current Daemonset Yaml files uses the new apiVersion.
If you are using and older Kubernetes version, manually grab a copy of your Daemonset Yaml file and replace the value of apiVersion from:
1
apiVersion: apps/v1
Copied!
to
1
apiVersion: extensions/v1beta1
Copied!
You can read more about this deprecation on Kubernetes v1.14 Changelog here:
​https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#deprecations​
Fluent Bit to Elasticsearch
Fluent Bit DaemonSet ready to be used with Elasticsearch on a normal Kubernetes Cluster:
1
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds.yaml
Copied!
Fluent Bit to Elasticsearch on Minikube
If you are using Minikube for testing purposes, use the following alternative DaemonSet manifest:
1
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds-minikube.yaml
Copied!
Installing with Helm Chart
​Helm is a package manager for Kubernetes and allows you to quickly deploy application packages into your running cluster. Fluent Bit is distributed via a helm chart found in the Fluent Helm Charts repo: https://github.com/fluent/helm-charts.
To add the Fluent Helm Charts repo use the following command
1
helm repo add fluent https://fluent.github.io/helm-charts
Copied!
To validate that the repo was added you can run helm search repo fluent to ensure the charts were added. The default chart can then be installed by running the following
1
helm install fluent-bit fluent/fluent-bit
Copied!
Default Values
The default chart values include configuration to read container logs, with Docker parsing, systemd logs apply Kubernetes metadata enrichment and finally output to an Elasticsearch cluster. You can modify the values file included https://github.com/fluent/helm-charts/blob/master/charts/fluent-bit/values.yaml to specify additional outputs, health checks, monitoring endpoints, or other configuration options.
Details
The default configuration of Fluent Bit makes sure of the following:
Consume all containers logs from the running Node.
The Tail input plugin will not append more than 5MB into the engine until they are flushed to the Elasticsearch backend. This limit aims to provide a workaround for backpressure scenarios.
The Kubernetes filter will enrich the logs with Kubernetes metadata, specifically labels and annotations. The filter only goes to the API Server when it cannot find the cached info, otherwise it uses the cache.
The default backend in the configuration is Elasticsearch set by the Elasticsearch Output Plugin. It uses the Logstash format to ingest the logs. If you need a different Index and Type, please refer to the plugin option and do your own adjustments.
There is an option called Retry_Limit set to False, that means if Fluent Bit cannot flush the records to Elasticsearch it will re-try indefinitely until it succeed.
Container Runtime Interface (CRI) parser
Fluent Bit by default assumes that logs are formatted by the Docker interface standard. However, when using CRI you can run into issues with malformed JSON if you do not modify the parser used. Fluent Bit includes a CRI log parser that can be used instead. An example of the parser is seen below:
1
# CRI Parser
2
[PARSER]
3
    # http://rubular.com/r/tjUt3Awgg4
4
    Name cri
5
    Format regex
6
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
7
    Time_Key    time
8
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Copied!
To use this parser change the Input section for your configuration from docker to cri
1
[INPUT]
2
    Name tail
3
    Path /var/log/containers/*.log
4
    Parser cri
5
    Tag kube.*
6
    Mem_Buf_Limit 5MB
7
    Skip_Long_Lines On
Copied!
Windows Deployment
Since v1.5.0, Fluent Bit supports deployment to Windows pods.
Log files overview
When deploying Fluent Bit to Kubernetes, there are three log files that you need to pay attention to.
C:\k\kubelet.err.log
This is the error log file from kubelet daemon running on host.
You will need to retain this file for future troubleshooting (to debug deployment failures etc.)
C:\var\log\containers\<pod>_<namespace>_<container>-<docker>.log
This is the main log file you need to watch. Configure Fluent Bit to follow this file.
It is actually a symlink to the Docker log file in C:\ProgramData\, with some additional metadata on its file name.
C:\ProgramData\Docker\containers\<docker>\<docker>.log
This is the log file produced by Docker.
Normally you don't directly read from this file, but you need to make sure that this file is visible from Fluent Bit.
Typically, your deployment yaml contains the following volume configuration.
1
spec:
2
  containers:
3
  - name: fluent-bit
4
    image: my-repo/fluent-bit:1.8.4
5
    volumeMounts:
6
    - mountPath: C:\k
7
      name: k
8
    - mountPath: C:\var\log
9
      name: varlog
10
    - mountPath: C:\ProgramData
11
      name: progdata
12
  volumes:
13
  - name: k
14
    hostPath:
15
      path: C:\k
16
  - name: varlog
17
    hostPath:
18
      path: C:\var\log
19
  - name: progdata
20
    hostPath:
21
      path: C:\ProgramData
Copied!
Configure Fluent Bit
Assuming the basic volume configuration described above, you can apply the following config to start logging. You can visualize this configuration here​
1
fluent-bit.conf: |
2
    [SERVICE]
3
      Parsers_File      C:\\fluent-bit\\parsers.conf
4
​
5
    [INPUT]
6
      Name              tail
7
      Tag               kube.*
8
      Path              C:\\var\\log\\containers\\*.log
9
      Parser            docker
10
      DB                C:\\fluent-bit\\tail_docker.db
11
      Mem_Buf_Limit     7MB
12
      Refresh_Interval  10
13
​
14
    [INPUT]
15
      Name              tail
16
      Tag               kubelet.err
17
      Path              C:\\k\\kubelet.err.log
18
      DB                C:\\fluent-bit\\tail_kubelet.db
19
​
20
    [FILTER]
21
      Name              kubernetes
22
      Match             kube.*
23
      Kube_URL          https://kubernetes.default.svc.cluster.local:443
24
​
25
    [OUTPUT]
26
      Name  stdout
27
      Match *
28
​
29
parsers.conf: |
30
    [PARSER]
31
        Name         docker
32
        Format       json
33
        Time_Key     time
34
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
35
        Time_Keep    On
Copied!
Mitigate unstable network on Windows pods
Windows pods often lack working DNS immediately after boot (#78479). To mitigate this issue, filter_kubernetes provides a built-in mechanism to wait until the network starts up:
DNS_Retries - Retries N times until the network start working (6)
DNS_Wait_Time - Lookup interval between network status checks (30)
By default, Fluent Bit waits for 3 minutes (30 seconds x 6 times). If it's not enough for you, tweak the configuration as follows.
1
[filter]
2
    Name kubernetes
3
    ...
4
    DNS_Retries 10
5
    DNS_Wait_Time 30
Copied!