Fluent Bit: Official Manual
SlackGitHubCommunity MeetingsSandbox and LabsWebinars
4.0
4.0
  • Fluent Bit Documentation
  • About
    • What is Fluent Bit?
    • A Brief History of Fluent Bit
    • Fluentd and Fluent Bit
    • License
    • Sandbox and Lab Resources
  • Concepts
    • Key Concepts
    • Buffering
    • Data Pipeline
      • Input
      • Parser
      • Filter
      • Buffer
      • Router
      • Output
  • Installation
    • Getting Started with Fluent Bit
    • Upgrade Notes
    • Supported Platforms
    • Requirements
    • Sources
      • Download Source Code
      • Build and Install
      • Build with Static Configuration
    • Linux Packages
      • Amazon Linux
      • Redhat / CentOS
      • Debian
      • Ubuntu
      • Raspbian / Raspberry Pi
    • Docker
    • Containers on AWS
    • Amazon EC2
    • Kubernetes
    • macOS
    • Windows
    • Yocto / Embedded Linux
    • Buildroot / Embedded Linux
  • Administration
    • Configuring Fluent Bit
      • YAML Configuration
        • Service
        • Parsers
        • Multiline Parsers
        • Pipeline
        • Plugins
        • Upstream Servers
        • Environment Variables
        • Includes
      • Classic mode
        • Format and Schema
        • Configuration File
        • Variables
        • Commands
        • Upstream Servers
        • Record Accessor
      • Unit Sizes
      • Multiline Parsing
    • Transport Security
    • Buffering and Storage
    • Backpressure
    • Scheduling and Retries
    • Networking
    • Memory Management
    • Monitoring
    • Multithreading
    • HTTP Proxy
    • Hot Reload
    • Troubleshooting
    • Performance Tips
    • AWS credentials
  • Local Testing
    • Validating your Data and Structure
    • Running a Logging Pipeline Locally
  • Data Pipeline
    • Pipeline Monitoring
    • Inputs
      • Collectd
      • CPU Log Based Metrics
      • Disk I/O Log Based Metrics
      • Docker Events
      • Docker Log Based Metrics
      • Dummy
      • Elasticsearch
      • Exec
      • Exec Wasi
      • Ebpf
      • Fluent Bit Metrics
      • Forward
      • Head
      • Health
      • HTTP
      • Kafka
      • Kernel Logs
      • Kubernetes Events
      • Memory Metrics
      • MQTT
      • Network I/O Log Based Metrics
      • NGINX Exporter Metrics
      • Node Exporter Metrics
      • OpenTelemetry
      • Podman Metrics
      • Process Exporter Metrics
      • Process Log Based Metrics
      • Prometheus Remote Write
      • Prometheus Scrape Metrics
      • Random
      • Serial Interface
      • Splunk
      • Standard Input
      • StatsD
      • Syslog
      • Systemd
      • Tail
      • TCP
      • Thermal
      • UDP
      • Windows Event Log
      • Windows Event Log (winevtlog)
      • Windows Exporter Metrics
    • Parsers
      • Configuring Parser
      • JSON
      • Regular Expression
      • LTSV
      • Logfmt
      • Decoders
    • Processors
      • Content Modifier
      • Labels
      • Metrics Selector
      • OpenTelemetry Envelope
      • Sampling
      • SQL
      • Filters as processors
      • Conditional processing
    • Filters
      • AWS Metadata
      • CheckList
      • ECS Metadata
      • Expect
      • GeoIP2 Filter
      • Grep
      • Kubernetes
      • Log to Metrics
      • Lua
      • Parser
      • Record Modifier
      • Modify
      • Multiline
      • Nest
      • Nightfall
      • Rewrite Tag
      • Standard Output
      • Sysinfo
      • Throttle
      • Type Converter
      • Tensorflow
      • Wasm
    • Outputs
      • Amazon CloudWatch
      • Amazon Kinesis Data Firehose
      • Amazon Kinesis Data Streams
      • Amazon S3
      • Azure Blob
      • Azure Data Explorer
      • Azure Log Analytics
      • Azure Logs Ingestion API
      • Counter
      • Dash0
      • Datadog
      • Dynatrace
      • Elasticsearch
      • File
      • FlowCounter
      • Forward
      • GELF
      • Google Chronicle
      • Google Cloud BigQuery
      • HTTP
      • InfluxDB
      • Kafka
      • Kafka REST Proxy
      • LogDNA
      • Loki
      • Microsoft Fabric
      • NATS
      • New Relic
      • NULL
      • Observe
      • OpenObserve
      • OpenSearch
      • OpenTelemetry
      • Oracle Log Analytics
      • PostgreSQL
      • Prometheus Exporter
      • Prometheus Remote Write
      • SkyWalking
      • Slack
      • Splunk
      • Stackdriver
      • Standard Output
      • Syslog
      • TCP and TLS
      • Treasure Data
      • Vivo Exporter
      • WebSocket
  • Stream Processing
    • Introduction to Stream Processing
    • Overview
    • Changelog
    • Getting Started
      • Fluent Bit + SQL
      • Check Keys and NULL values
      • Hands On 101
  • Fluent Bit for Developers
    • C Library API
    • Ingest Records Manually
    • Golang Output Plugins
    • WASM Filter Plugins
    • WASM Input Plugins
    • Developer guide for beginners on contributing to Fluent Bit
Powered by GitBook
On this page
  • Get started
  • Build a Go plugin
  • Run Fluent Bit with the new plugin
  • Configuration file
  • Plugins configuration file
  • Main configuration file
  • Run using a configuration file

Was this helpful?

Export as PDF
  1. Fluent Bit for Developers

Golang Output Plugins

Fluent Bit supports integration of Golang plugins built as shared objects for output plugins only. The interface for the Golang plugins is currently under development but is functional.

Get started

Compile Fluent Bit with Golang support:

cd build/
cmake -DFLB_DEBUG=On -DFLB_PROXY_GO=On ../
make

Once compiled, you can see the new -e option in the binary which stands for external plugin.

$ bin/fluent-bit -h
Usage: fluent-bit [OPTION]

Available Options
  -c  --config=FILE    specify an optional configuration file
  -d, --daemon        run Fluent Bit in background mode
  -f, --flush=SECONDS    flush timeout in seconds (default: 1)
  -i, --input=INPUT    set an input
  -m, --match=MATCH    set plugin match, same as '-p match=abc'
  -o, --output=OUTPUT    set an output
  -p, --prop="A=B"    set plugin configuration property
  -e, --plugin=FILE    load an external plugin (shared lib)
  ...

Build a Go plugin

The fluent-bit-go package is available to assist developers in creating Go plugins.

A minimum Go plugin looks like the following:

package main

import "github.com/fluent/fluent-bit-go/output"

//export FLBPluginRegister
func FLBPluginRegister(def unsafe.Pointer) int {
    // Gets called only once when the plugin.so is loaded
    return output.FLBPluginRegister(def, "gstdout", "Stdout GO!")
}

//export FLBPluginInit
func FLBPluginInit(plugin unsafe.Pointer) int {
    // Gets called only once for each instance you have configured.
    return output.FLB_OK
}

//export FLBPluginFlushCtx
func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int {
    // Gets called with a batch of records to be written to an instance.
    return output.FLB_OK
}

//export FLBPluginExit
func FLBPluginExit() int {
    return output.FLB_OK
}

func main() {
}

The previous code is a template to write an output plugin. It's important to keep the package name as main and add an explicit main() function. This is a requirement as the code will be build as a shared library.

To build the code, use the following line:

go build -buildmode=c-shared -o out_gstdout.so out_gstdout.go

Once built, a shared library called out_gstdout.so will be available. Confirm the final .so file is as expected. When you use the ldd over the library should see something similar to this:

$ ldd out_gstdout.so
    linux-vdso.so.1 =>  (0x00007fff561dd000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc4aeef0000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc4aeb27000)
    /lib64/ld-linux-x86-64.so.2 (0x000055751a4fd000)

Run Fluent Bit with the new plugin

bin/fluent-bit -e /path/to/out_gstdout.so -i cpu -o gstdout

Configuration file

Fluent Bit can load and run Golang plugins using two configuration files.

  • Plugins configuration file

Plugins configuration file

Key
Description

Path

A path for a Golang plugin.

Plugin file example

[PLUGINS]
    Path /path/to/out_gstdout.so

Main configuration file

The keys for Golang plugin available as of this version are described in the following table:

Key
Description

Plugins_file

Main configuration file example

The following is an example of a main configuration file.

[SERVICE]
    plugins_file /path/to/plugins.conf

[INPUT]
    Name dummy

[OUTPUT]
    Name gstdout

Config key constraint

The following configuration keys are reserved by Fluent Bit and must not be used by a custom plugin:

  • alias

  • host

  • ipv6

  • listen

  • log_level

  • log_suppress_interval

  • match

  • match_regex

  • mem_buf_limit

  • port

  • retry_limit

  • routable

  • storage.pause_on_chunks_overlimit

  • storage.total_limit_size

  • storage.type

  • tag

  • threaded

  • tls

  • tls.ca_file

  • tls.ca_path

  • tls.crt_file

  • tls.debug

  • tls.key_file

  • tls.key_passwd

  • tls.verify

  • tls.vhost

  • workers

Run using a configuration file

You can load a main configuration file using -c option. You don't need to specify a plugins configuration file from command line.

fluent-bit -c fluent-bit.conf

Last updated 5 days ago

Was this helpful?

Path for a plugins configuration file. A plugins configuration file let you define paths for external plugins. For example, .

https://github.com/fluent/fluent-bit-go
Main configuration file
see here