Loki
​Loki is multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate.
The Fluent Bit
loki built-in output plugin allows you to send your log or events to a Loki service. It supports data enrichment with Kubernetes labels, custom label keys and Tenant ID within others.Be aware there is a separate Golang output plugin provided by Grafana with different configuration options.
Configuration Parameters
Key
Description
Default
host
Loki hostname or IP address. Do not include the subpath, i.e.
loki/api/v1/push, but just the base hostname/URL.127.0.0.1
port
Loki TCP port
3100
http_user
Set HTTP basic authentication user name
​
http_passwd
Set HTTP basic authentication password
​
tenant_id
Tenant ID used by default to push logs to Loki. If omitted or empty it assumes Loki is running in single-tenant mode and no X-Scope-OrgID header is sent.
​
labels
Stream labels for API request. It can be multiple comma separated of strings specifying
key=value pairs. In addition to fixed parameters, it also allows to add custom record keys (similar to label_keys property). More details in the Labels section.job=fluentbit
label_keys
Optional list of record keys that will be placed as stream labels. This configuration property is for records key only. More details in the Labels section.
​
remove_keys
Optional list of keys to remove.
​
drop_single_key
If set to true and after extracting labels only a single key remains, the log line sent to Loki will be the value of that key in line_format.
off
line_format
Format to use when flattening the record to a log line. Valid values are
json or key_value. If set to json, the log line sent to Loki will be the Fluent Bit record dumped as JSON. If set to key_value, the log line will be each item in the record concatenated together (separated by a single space) in the format.json
auto_kubernetes_labels
If set to true, it will add all Kubernetes labels to the Stream labels
off
tenant_id_key
Specify the name of the key from the original record that contains the Tenant ID. The value of the key is set as
X-Scope-OrgID of HTTP header. It is useful to set Tenant ID dynamically.​
Labels
Loki store the record logs inside Streams, a stream is defined by a set of labels, at least one label is required.
Fluent Bit implements a flexible mechanism to set labels by using fixed key/value pairs of text but also allowing to set as labels certain keys that exists as part of the records that are being processed. Consider the following JSON record (pretty printed for readability):
1
{
2
"key": 1,
3
"sub": {
4
"stream": "stdout",
5
"id": "some id"
6
},
7
"kubernetes": {
8
"labels": {
9
"team": "Santiago Wanderers"
10
}
11
}
12
}
Copied!
If you decide that your Loki Stream will be composed by two labels called
job and the value of the record key called stream , your labels configuration properties might look as follows:1
[OUTPUT]
2
name loki
3
match *
4
labels job=fluentbit, $sub['stream']
Copied!
As you can see the label
job has the value fluentbit and the second label is configured to access the nested map called sub targeting the value of the key stream . Note that the second label name must starts with a $, that means that's a Record Accessor pattern so it provide you the ability to retrieve values from nested maps by using the key names.When processing above's configuration, internally the ending labels for the stream in question becomes:
1
job="fluentbit", stream="stdout"
Copied!
Another feature of Labels management is the ability to provide custom key names, using the same record accessor pattern we can specify the key name manually and let the value to be populated automatically at runtime, e.g:
1
[OUTPUT]
2
name loki
3
match *
4
labels job=fluentbit, mystream=$sub['stream']
Copied!
When processing that new configuration, the internal labels will be:
1
job="fluentbit", mystream="stdout"
Copied!
Using the label_keys property
label_keys propertyThe additional configuration property called
label_keys allow to specify multiple record keys that needs to be placed as part of the outgoing Stream Labels, yes, this is a similar feature than the one explained above in the labels property. Consider this as another way to set a record key in the Stream, but with the limitation that you cannot use a custom name for the key value.The following configuration examples generate the same Stream Labels:
1
[OUTPUT]
2
name loki
3
match *
4
labels job=fluentbit
5
label_keys $sub['stream']
Copied!
the above configuration accomplish the same than this one:
1
[OUTPUT]
2
name loki
3
match *
4
labels job=fluentbit, $sub['stream']
Copied!
both will generate the following Streams label:
1
job="fluentbit", stream="stdout"
Copied!
Kubernetes & Labels
Note that if you are running in a Kubernetes environment, you might want to enable the option
auto_kubernetes_labels which will auto-populate the streams with the Pod labels for you. Consider the following configuration:1
[OUTPUT]
2
name loki
3
match *
4
labels job=fluentbit
5
auto_kubernetes_labels on
Copied!
Based in the JSON example provided above, the internal stream labels will be:
1
job="fluentbit", team="Santiago Wanderers"
Copied!
Networking and TLS Configuration
This plugin inherit core Fluent Bit features to customize the network behavior and optionally enable TLS in the communication channel. For more details about the specific options available refer to the following articles:
Note that all options mentioned in the articles above must be enabled in the plugin configuration in question.
Fluent Bit + Grafana Cloud
Fluent Bit supports sending logs (and metrics) to Grafana Cloud by providing the appropriate URL and ensuring TLS is enabled.
An example configuration - make sure to set the credentials and ensure the host URL matches the correct one for your deployment:
1
[OUTPUT]
2
Name loki
3
Match *
4
Host logs-prod-eu-west-0.grafana.net
5
port 443
6
tls on
7
tls.verify on
8
http_user XXX
9
http_passwd XXX
Copied!
Getting Started
The following configuration example, will emit a dummy example record and ingest it on Loki . Copy and paste the following content into a file called
out_loki.conf:1
[SERVICE]
2
flush 1
3
log_level info
4
​
5
[INPUT]
6
name dummy
7
dummy {"key": 1, "sub": {"stream": "stdout", "id": "some id"}, "kubernetes": {"labels": {"team": "Santiago Wanderers"}}}
8
samples 1
9
​
10
[OUTPUT]
11
name loki
12
match *
13
host 127.0.0.1
14
port 3100
15
labels job=fluentbit
16
label_keys $sub['stream']
17
auto_kubernetes_labels on
Copied!
run Fluent Bit with the new configuration file:
1
$ fluent-bit -c out_loki.conf
Copied!
Fluent Bit output:
1
Fluent Bit v1.7.0
2
* Copyright (C) 2019-2020 The Fluent Bit Authors
3
* Copyright (C) 2015-2018 Treasure Data
4
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
5
* https://fluentbit.io
6
​
7
[2020/10/14 20:57:45] [ info] [engine] started (pid=809736)
8
[2020/10/14 20:57:45] [ info] [storage] version=1.0.6, initializing...
9
[2020/10/14 20:57:45] [ info] [storage] in-memory
10
[2020/10/14 20:57:45] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
11
[2020/10/14 20:57:45] [ info] [output:loki:loki.0] configured, hostname=127.0.0.1:3100
12
[2020/10/14 20:57:45] [ info] [sp] stream processor started
13
[2020/10/14 20:57:46] [debug] [http] request payload (272 bytes)
14
[2020/10/14 20:57:46] [ info] [output:loki:loki.0] 127.0.0.1:3100, HTTP status=204
Copied!
Last modified 3mo ago