Fluent Bit: Official Manual
SlackGitHubCommunity MeetingsSandbox and LabsWebinars
1.5
1.5
  • Fluent Bit v1.5 Documentation
  • About
    • What is Fluent Bit ?
    • A Brief History of Fluent Bit
    • Fluentd & Fluent Bit
    • License
  • Concepts
    • Key Concepts
    • Buffering
    • Data Pipeline
      • Input
      • Parser
      • Filter
      • Buffer
      • Router
      • Output
  • Installation
    • Upgrade Notes
    • Supported Platforms
    • Requirements
    • Sources
      • Download Source Code
      • Build and Install
      • Build with Static Configuration
    • Linux Packages
      • Amazon Linux
      • Redhat / CentOS
      • Debian
      • Ubuntu
      • Raspbian / Raspberry Pi
    • Docker
    • Containers on AWS
    • Amazon EC2
    • Kubernetes
    • Yocto / Embedded Linux
    • Windows
  • Administration
    • Configuring Fluent Bit
      • Format and Schema
      • Configuration File
      • Variables
      • Commands
      • Upstream Servers
      • Unit Sizes
      • Record Accessor
    • Security
    • Buffering & Storage
    • Backpressure
    • Scheduling and Retries
    • Networking
    • Memory Management
    • Monitoring
    • Dump Internals / Signal
  • Local Testing
    • Validating your Data and Structure
    • Running a Logging Pipeline Locally
  • Data Pipeline
    • Inputs
      • Collectd
      • CPU Metrics
      • Disk I/O Metrics
      • Docker Events
      • Dummy
      • Exec
      • Forward
      • Head
      • Health
      • Kernel Logs
      • Memory Metrics
      • MQTT
      • Network I/O Metrics
      • Process
      • Random
      • Serial Interface
      • Standard Input
      • Syslog
      • Systemd
      • Tail
      • TCP
      • Thermal
      • Windows Event Log
    • Parsers
      • JSON
      • Regular Expression
      • LTSV
      • Logfmt
      • Decoders
    • Filters
      • AWS Metadata
      • Expect
      • Grep
      • Kubernetes
      • Lua
      • Parser
      • Record Modifier
      • Rewrite Tag
      • Standard Output
      • Throttle
      • Nest
      • Modify
    • Outputs
      • Amazon CloudWatch
      • Azure
      • BigQuery
      • Counter
      • Datadog
      • Elasticsearch
      • File
      • FlowCounter
      • Forward
      • GELF
      • HTTP
      • InfluxDB
      • Kafka
      • Kafka REST Proxy
      • LogDNA
      • NATS
      • New Relic
      • NULL
      • PostgreSQL
      • Stackdriver
      • Standard Output
      • Splunk
      • Syslog
      • TCP & TLS
      • Treasure Data
  • Stream Processing
    • Introduction to Stream Processing
    • Overview
    • Changelog
    • Getting Started
      • Fluent Bit + SQL
      • Check Keys and NULL values
      • Hands On! 101
  • Fluent Bit for Developers
    • C Library API
    • Ingest Records Manually
    • Golang Output Plugins
    • Developer guide for beginners on contributing to Fluent Bit
Powered by GitBook
On this page
  • Configuration Parameters
  • Getting Started
  • Command Line
  • Configuration File
  • Lua Script Filter API
  • Callback Prototype
  • Code Examples
  • Number Type
  • Protected Mode

Was this helpful?

Export as PDF
  1. Data Pipeline
  2. Filters

Lua

Last updated 4 years ago

Was this helpful?

Lua Filter allows you to modify the incoming records using custom Scripts.

Due to the necessity to have a flexible filtering mechanism, now is possible to extend Fluent Bit capabilities writing simple filters using Lua programming language. A Lua based filter takes two steps:

  • Configure the Filter in the main configuration

  • Prepare a Lua script that will be used by the Filter

Configuration Parameters

The plugin supports the following configuration parameters:

Key

Description

Script

Path to the Lua script that will be used.

Call

Lua function name that will be triggered to do filtering. It's assumed that the function is declared inside the Script defined above.

Type_int_key

If these keys are matched, the fields are converted to integer. If more than one key, delimit by space

Protected_mode

If enabled, Lua script will be executed in protected mode. It prevents to crash when invalid Lua script is executed. Default is true.

Getting Started

In order to test the filter, you can run the plugin from the command line or through the configuration file. The following examples uses the input plugin for data ingestion, invoke Lua filter using the script and calls the function which only print the same information to the standard output:

Command Line

From the command line you can use the following options:

$ fluent-bit -i dummy -F lua -p script=test.lua -p call=cb_print -m '*' -o null

Configuration File

In your main configuration file append the following Input, Filter & Output sections:

[INPUT]
    Name   dummy

[FILTER]
    Name    lua
    Match   *
    script  test.lua
    call    cb_print

[OUTPUT]
    Name   null
    Match  *

Lua Script Filter API

The life cycle of a filter have the following steps:

  • Upon Tag matching by filter_lua, it may process or bypass the record.

  • If filter_lua accepts the record, it will invoke the function defined in the call property which basically is the name of a function defined in the Lua script.

  • Invoke Lua function passing each record in JSON format.

  • Upon return, validate return value and take some action (described above)

Callback Prototype

The Lua script can have one or multiple callbacks that can be used by filter_lua, it prototype is as follows:

function cb_print(tag, timestamp, record)
   return code, timestamp, record
end

Function Arguments

name

description

tag

Name of the tag associated with the incoming record.

timestamp

Unix timestamp with nanoseconds associated with the incoming record. The original format is a double (seconds.nanoseconds)

record

Lua table with the record content

Return Values

Each callback must return three values:

name

data type

description

code

integer

The code return value represents the result and further action that may follows. If code equals -1, means that filter_lua must drop the record. If code equals 0 the record will not be modified, otherwise if code equals 1, means the original timestamp and record have been modified so it must be replaced by the returned values from timestamp (second return value) and record (third return value). If code equals 2, means the original timestamp is not modified and the record has been modified so it must be replaced by the returned values from record (third return value). The code 2 is supported from v1.4.3.

timestamp

double

If code equals 1, the original record timestamp will be replaced with this new value.

record

table

if code equals 1, the original record information will be replaced with this new value. Note that the format of this value must be a valid Lua table.

Code Examples

For functional examples of this interface, please refer to the code samples provided in the source code of the project located here:

Number Type

In Lua, Fluent Bit treats number as double. It means an integer field (e.g. IDs, log levels) will be converted double. To avoid type conversion, Type_int_key property is available.

Protected Mode

Fluent Bit supports protected mode to prevent crash when executes invalid Lua script. See also .

Lua
dummy
test.lua
cb_print()
https://github.com/fluent/fluent-bit/tree/master/scripts
Error Handling in Application Code