Record Accessor

Fluent Bit works internally with structured records and it can be composed of an unlimited number of keys and values. Values can be anything like a number, string, array, or a map.
Having a way to select a specific part of the record is critical for certain core functionalities or plugins, this feature is called Record Accessor.
consider Record Accessor a simple grammar to specify record content and other miscellaneous values.
Format
A record accessor rule starts with the character $. Using the structured content above as an example the following table describes how to access a record:
1
{
2
  "log": "some message",
3
  "stream": "stdout",
4
  "labels": {
5
     "color": "blue", 
6
     "unset": null,
7
     "project": {
8
         "env": "production"
9
      }
10
  }
11
}
Copied!
The following table describe some accessing rules and the expected returned value:
Format
Accessed Value
$log
"some message"
$labels['color']
"blue"
$labels['project']['env']
"production"
$labels['unset']
null
$labels['undefined']
​
If the accessor key does not exist in the record like the last example $labels['undefined'] , the operation is simply omitted, no exception will occur.
Usage Example
The feature is enabled on a per plugin basis, not all plugins enable this feature. As an example consider a configuration that aims to filter records using grep that only matches where labels have a color blue:
1
[SERVICE]
2
    flush        1
3
    log_level    info
4
    parsers_file parsers.conf
5
​
6
[INPUT]
7
    name      tail
8
    path      test.log
9
    parser    json
10
​
11
[FILTER]
12
    name      grep
13
    match     *
14
    regex     $labels['color'] ^blue$
15
​
16
[OUTPUT]
17
    name      stdout
18
    match     *
19
    format    json_lines
Copied!
The file content to process in test.log is the following:
1
{"log": "message 1", "labels": {"color": "blue"}}
2
{"log": "message 2", "labels": {"color": "red"}}
3
{"log": "message 3", "labels": {"color": "green"}}
4
{"log": "message 4", "labels": {"color": "blue"}}
Copied!
Running Fluent Bit with the configuration above the output will be:
1
$ bin/fluent-bit -c fluent-bit.conf 
2
Fluent Bit v1.x.x
3
* Copyright (C) 2019-2020 The Fluent Bit Authors
4
* Copyright (C) 2015-2018 Treasure Data
5
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
6
* https://fluentbit.io
7
​
8
[2020/09/11 16:11:07] [ info] [engine] started (pid=1094177)
9
[2020/09/11 16:11:07] [ info] [storage] version=1.0.5, initializing...
10
[2020/09/11 16:11:07] [ info] [storage] in-memory
11
[2020/09/11 16:11:07] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
12
[2020/09/11 16:11:07] [ info] [sp] stream processor started
13
[2020/09/11 16:11:07] [ info] inotify_fs_add(): inode=55716713 watch_fd=1 name=test.log
14
{"date":1599862267.483684,"log":"message 1","labels":{"color":"blue"}}
15
{"date":1599862267.483692,"log":"message 4","labels":{"color":"blue"}}
Copied!

Upstream Servers

YAML Configuration

Last modified 3mo ago

Export as PDF

Copy link

Contents

Format

Usage Example