Kubernetes Production Grade Log Processor
- Process Kubernetes containers logs from the file system or Systemd/Journald.
- Enrich logs with Kubernetes Metadata.
- Centralize your logs in third party storage services like Elasticsearch, InfluxDB, HTTP, etc.
Before getting started it is important to understand how Fluent Bit will be deployed. Kubernetes manages a cluster of nodes, so our log agent tool will need to run on every node to collect logs from every POD, hence Fluent Bit is deployed as a DaemonSet (a POD that runs on every node of the cluster).
When Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich each entry with the following information (metadata):
- Pod Name
- Pod ID
- Container Name
- Container ID
To obtain this information, a built-in filter plugin called kubernetes talks to the Kubernetes API Server to retrieve relevant information such as the pod_id, labels and annotations, other fields such as pod_name, container_id and container_name are retrieved locally from the log file names. All of this is handled automatically, no intervention is required from a configuration aspect.
The recommended way to deploy Fluent Bit is with the official Helm Chart: https://github.com/fluent/helm-charts
If you are using Red Hat OpenShift you will also need to set up security context constraints (SCC):
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-openshift-security-context-constraints.yaml
To add the Fluent Helm Charts repo use the following command
helm repo add fluent https://fluent.github.io/helm-charts
To validate that the repo was added you can run
helm search repo fluentto ensure the charts were added. The default chart can then be installed by running the following
helm upgrade --install fluent-bit fluent/fluent-bit
The default chart values include configuration to read container logs, with Docker parsing, systemd logs apply Kubernetes metadata enrichment and finally output to an Elasticsearch cluster. You can modify the values file included https://github.com/fluent/helm-charts/blob/master/charts/fluent-bit/values.yaml to specify additional outputs, health checks, monitoring endpoints, or other configuration options.
The default configuration of Fluent Bit makes sure of the following:
- Consume all containers logs from the running Node.
- The Kubernetes filter will enrich the logs with Kubernetes metadata, specifically labels and annotations. The filter only goes to the API Server when it cannot find the cached info, otherwise it uses the cache.
- The default backend in the configuration is Elasticsearch set by the Elasticsearch Output Plugin. It uses the Logstash format to ingest the logs. If you need a different Index and Type, please refer to the plugin option and do your own adjustments.
- There is an option called Retry_Limit set to False, that means if Fluent Bit cannot flush the records to Elasticsearch it will re-try indefinitely until it succeed.
Fluent Bit by default assumes that logs are formatted by the Docker interface standard. However, when using CRI you can run into issues with malformed JSON if you do not modify the parser used. Fluent Bit includes a CRI log parser that can be used instead. An example of the parser is seen below:
# CRI Parser
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
To use this parser change the Input section for your configuration from
Since v1.5.0, Fluent Bit supports deployment to Windows pods.
When deploying Fluent Bit to Kubernetes, there are three log files that you need to pay attention to.
- This is the error log file from kubelet daemon running on host.
- You will need to retain this file for future troubleshooting (to debug deployment failures etc.)
- This is the main log file you need to watch. Configure Fluent Bit to follow this file.
- It is actually a symlink to the Docker log file in
C:\ProgramData\, with some additional metadata on its file name.
- This is the log file produced by Docker.
- Normally you don't directly read from this file, but you need to make sure that this file is visible from Fluent Bit.
Typically, your deployment yaml contains the following volume configuration.
- name: fluent-bit
- mountPath: C:\k
- mountPath: C:\var\log
- mountPath: C:\ProgramData
- name: k
- name: varlog
- name: progdata
DNS_Retries- Retries N times until the network start working (6)
DNS_Wait_Time- Lookup interval between network status checks (30)
By default, Fluent Bit waits for 3 minutes (30 seconds x 6 times). If it's not enough for you, tweak the configuration as follows.