Lua - Fluent Bit: Official Manual

The Lua filter allows you to modify the incoming records (even split one record into multiple records) using custom Lua scripts.
Due to the necessity to have a flexible filtering mechanism, it is now possible to extend Fluent Bit capabilities by writing custom filters using Lua programming language. A Lua-based filter takes two steps:
1.Configure the Filter in the main configuration
2.Prepare a Lua script that will be used by the Filter
Configuration Parameters
The plugin supports the following configuration parameters:
Key
Description
script
Path to the Lua script that will be used. This can be a relative path against the main configuration file.
call
Lua function name that will be triggered to do filtering. It's assumed that the function is declared inside the script parameter defined above.
type_int_key
If these keys are matched, the fields are converted to integer. If more than one key, delimit by space. Note that starting from Fluent Bit v1.6 integer data types are preserved and not converted to double as in previous versions.
type_array_key
If these keys are matched, the fields are handled as array. If more than one key, delimit by space. It is useful the array can be empty.
protected_mode
If enabled, Lua script will be executed in protected mode. It prevents Fluent Bit from crashing when invalid Lua script is executed or the triggered Lua function throws exceptions. Default is true.
time_as_table
By default when the Lua script is invoked, the record timestamp is passed as a floating number which might lead to precision loss when it is converted back. If you desire timestamp precision, enabling this option will pass the timestamp as a Lua table with keys sec for seconds since epoch and nsec for nanoseconds.
code
Inline LUA code instead of loading from a path via script.
Getting Started
In order to test the filter, you can run the plugin from the command line or through the configuration file. The following examples use the dummy input plugin for data ingestion, invoke Lua filter using the test.lua script and call the cb_print() function which only prints the same information to the standard output:
Command Line
From the command line you can use the following options:
1
$ fluent-bit -i dummy -F lua -p script=test.lua -p call=cb_print -m '*' -o null
Copied!
Configuration File
In your main configuration file append the following Input, Filter & Output sections:
1
[INPUT]
2
    Name    dummy
3
​
4
[FILTER]
5
    Name    lua
6
    Match   *
7
    script  test.lua
8
    call    cb_print
9
​
10
[OUTPUT]
11
    Name    null
12
    Match   *
Copied!
Lua Script Filter API
The life cycle of a filter have the following steps:
1.Upon Tag matching by this filter, it may process or bypass the record.
2.If tag matched, it will accept the record and invoke the function defined in the call property which basically is the name of a function defined in the Lua script.
3.Invoke Lua function and pass each record in JSON format.
4.Upon return, validate return value and continue the pipeline.
Callback Prototype
The Lua script can have one or multiple callbacks that can be used by this filter. The function prototype is as follows:
1
function cb_print(tag, timestamp, record)
2
    ...
3
    return code, timestamp, record
4
end
Copied!
Function Arguments
name
description
tag
Name of the tag associated with the incoming record.
timestamp
Unix timestamp with nanoseconds associated with the incoming record. The original format is a double (seconds.nanoseconds)
record
Lua table with the record content
Return Values
Each callback must return three values:
name
data type
description
code
integer
The code return value represents the result and further action that may follows. If code equals -1, means that the record will be dropped. If code equals 0, the record will not be modified, otherwise if code equals 1, means the original timestamp and record have been modified so it must be replaced by the returned values from timestamp (second return value) and record (third return value). If code equals 2, means the original timestamp is not modified and the record has been modified so it must be replaced by the returned values from record (third return value). The code 2 is supported from v1.4.3.
timestamp
double
If code equals 1, the original record timestamp will be replaced with this new value.
record
table
If code equals 1, the original record information will be replaced with this new value. Note that the record value must be a valid Lua table. This value can be an array of tables (i.e., array of objects in JSON format), and in that case the input record is effectively split into multiple records. (see below for more details)
Code Examples
For functional examples of this interface, please refer to the code samples provided in the source code of the project located here:
​https://github.com/fluent/fluent-bit/tree/master/scripts​
Inline configuration
The Fluent Bit smoke tests include examples to verify during CI.
1
service:
2
    flush:           1
3
    daemon:          off
4
    log_level:       info
5
​
6
pipeline:
7
    inputs:
8
        - random:
9
            tag:           test
10
            samples:       10
11
​
12
    filters:
13
        - lua:
14
            match:         "*"
15
            call:          append_tag
16
            code:          |
17
                function append_tag(tag, timestamp, record)
18
                   new_record = record
19
                   new_record["tag"] = tag
20
                   return 1, timestamp, new_record
21
                end
22
​
23
    outputs:
24
        - stdout:
25
            match:         "*"
Copied!
In classic mode:
1
[SERVICE]
2
	flush 1
3
	daemon off
4
	log_level debug
5
​
6
[INPUT]
7
	Name random
8
	Tag test
9
	Samples 10
10
​
11
[FILTER]
12
	Name Lua
13
	Match *
14
	call append_tag
15
	code function append_tag(tag, timestamp, record) new_record = record new_record["tag"] = tag return 1, timestamp, new_record end
16
​
17
[OUTPUT]
18
	Name stdout
19
	Match *
Copied!
Environment variable processing
As an example that combines a bit of LUA processing with the Kubernetes filter that demonstrates using environment variables with LUA regex and substitutions.
Kubernetes pods generally have various environment variables set by the infrastructure automatically which may contain useful information.
In this example, we want to extract part of the Kubernetes cluster API name.
The environment variable is set like so: KUBERNETES_SERVICE_HOST: api.sandboxbsh-a.project.domain.com
We want to extract the sandboxbsh name and add it to our record as a special key.
1
      [FILTER]
2
          Name                lua
3
          Alias               filter-iots-lua
4
          Match               iots_thread.*
5
          Script              filters.lua
6
          Call                set_landscape_deployment
7
​
8
  filters.lua: |
9
    -- Use a Lua function to create some additional entries based
10
    -- on substrings from the kubernetes properties.
11
    function set_landscape_deployment(tag, timestamp, record)
12
        local landscape = os.getenv("KUBERNETES_SERVICE_HOST")
13
        if landscape then
14
            -- Strip the landscape name from this field, KUBERNETES_SERVICE_HOST
15
            -- Should be of this format
16
            -- api.sandboxbsh-a.project.domain.com
17
            -- Take off the leading "api."
18
            -- sandboxbsh-a.project.domain.com
19
            --print("landscape1:" .. landscape)
20
            landscape = landscape:gsub("^[^.]+.", "")
21
            --print("landscape2:" .. landscape)
22
            -- Take off everything including and after the - in the cluster name
23
            -- sandboxbsh
24
            landscape = landscape:gsub("-.*quot;, "")
25
            -- print("landscape3:" .. landscape)
26
            record["iot_landscape"] = landscape
27
        end
28
        -- 2 - replace existing record with this update
29
        return 2, timestamp, record
30
    end
Copied!
Number Type
+Lua treats number as double. It means an integer field (e.g. IDs, log levels) will be converted double. To avoid type conversion, The type_int_key property is available.
Protected Mode
Fluent Bit supports protected mode to prevent crash when executes invalid Lua script. See also Error Handling in Application Code.
Record Split
The Lua callback function can return an array of tables (i.e., array of records) in its third record return value. With this feature, the Lua filter can split one input record into multiple records according to custom logic.
For example:
Lua script
1
function cb_split(tag, timestamp, record)
2
    if record["x"] ~= nil then
3
        return 2, timestamp, record["x"]
4
    else
5
        return 2, timestamp, record
6
    end
7
end
Copied!
Configuration
1
[Input]
2
    Name    stdin
3
​
4
[Filter]
5
    Name    lua
6
    Match   *
7
    script  test.lua
8
    call    cb_split
9
​
10
[Output]
11
    Name    stdout
12
    Match   *
Copied!
Input
1
{"x": [ {"a1":"aa", "z1":"zz"}, {"b1":"bb", "x1":"xx"}, {"c1":"cc"} ]}
2
{"x": [ {"a2":"aa", "z2":"zz"}, {"b2":"bb", "x2":"xx"}, {"c2":"cc"} ]}
3
{"a3":"aa", "z3":"zz", "b3":"bb", "x3":"xx", "c3":"cc"}
Copied!
Output
1
[0] stdin.0: [1538435928.310583591, {"a1"=>"aa", "z1"=>"zz"}]
2
[1] stdin.0: [1538435928.310583591, {"x1"=>"xx", "b1"=>"bb"}]
3
[2] stdin.0: [1538435928.310583591, {"c1"=>"cc"}]
4
[3] stdin.0: [1538435928.310588359, {"z2"=>"zz", "a2"=>"aa"}]
5
[4] stdin.0: [1538435928.310588359, {"b2"=>"bb", "x2"=>"xx"}]
6
[5] stdin.0: [1538435928.310588359, {"c2"=>"cc"}]
7
[6] stdin.0: [1538435928.310589790, {"z3"=>"zz", "x3"=>"xx", "c3"=>"cc", "a3"=>"aa", "b3"=>"bb"}]
Copied!
See also Fluent Bit: PR 811.

Kubernetes

Parser

Last modified 21d ago

Export as PDF

Copy link

Contents

Configuration Parameters

Getting Started

Command Line

Configuration File

Lua Script Filter API