Fluent Bit library is written in C language and can be used from any C or C++ application. Before digging into the specification it is recommended to understand the workflow involved in the runtime.
Workflow
Fluent Bit runs as a service, meaning that the API exposed for developers provide interfaces to create and manage a context, specify inputs/outputs, set configuration parameters and set routing paths for the event/records. A typical usage of the library involves:
Create library instance/context and set properties.
Enable input plugin(s) and set properties.
Enable output plugin(s) and set properties.
Start the library runtime.
Optionally ingest records manually.
Stop the library runtime.
Destroy library instance/context.
Data Types
Starting from Fluent Bit v0.9, there is only one data type exposed by the library, by convention prefixed with flb_.
API Reference
Library Context Creation
As described earlier, the first step to use the library is to create a context of it, for the purpose the function flb_create() is used.
Prototype
Return Value
On success, flb_create() returns the library context; on error, it returns NULL.
Usage
Set Service Properties
Using the flb_service_set() function is possible to set context properties.
Prototype
Return Value
On success it returns 0; on error it returns a negative number.
Usage
The flb_service_set() allows to set one or more properties in a key/value string mode, e.g:
The above example specified the values for the properties Flush , note that the value is always a string (char *) and once there is no more parameters a NULL argument must be added at the end of the list.
Enable Input Plugin Instance
When built, library contains a certain number of built-in input plugins. In order to enable an input plugin, the function flb_input() is used to create an instance of it.
For plugins, an instance means a context of the plugin enabled. You can create multiples instances of the same plugin.
Prototype
The argument ctx represents the library context created by flb_create(), then name is the name of the input plugin that is required to enable.
The third argument data can be used to pass a custom reference to the plugin instance, this is mostly used by custom or third party plugins, for generic plugins passing NULL is OK.
Return Value
On success, flb_input() returns an integer value >= zero (similar to a file descriptor); on error, it returns a negative number.
Usage
Set Input Plugin Properties
A plugin instance created through flb_input(), may provide some configuration properties. Using the flb_input_set() function is possible to set these properties.
Prototype
Return Value
On success it returns 0; on error it returns a negative number.
Usage
The flb_input_set() allows to set one or more properties in a key/value string mode, e.g:
The argument ctx represents the library context created by flb_create(). The above example specified the values for the properties tag and ssl, note that the value is always a string (char *) and once there is no more parameters a NULL argument must be added at the end of the list.
The properties allowed per input plugin are specified on each specific plugin documentation.
Enable Output Plugin Instance
When built, library contains a certain number of built-in output plugins. In order to enable an output plugin, the function flb_output() is used to create an instance of it.
For plugins, an instance means a context of the plugin enabled. You can create multiples instances of the same plugin.
Prototype
The argument ctx represents the library context created by flb_create(), then name is the name of the output plugin that is required to enable.
The third argument data can be used to pass a custom reference to the plugin instance, this is mostly used by custom or third party plugins, for generic plugins passing NULL is OK.
Return Value
On success, flb_output() returns the output plugin instance; on error, it returns a negative number.
Usage
Set Output Plugin Properties
A plugin instance created through flb_output(), may provide some configuration properties. Using the flb_output_set() function is possible to set these properties.
Prototype
Return Value
On success it returns an integer value >= zero (similar to a file descriptor); on error it returns a negative number.
Usage
The flb_output_set() allows to set one or more properties in a key/value string mode, e.g:
The argument ctx represents the library context created by flb_create(). The above example specified the values for the properties tag and ssl, note that the value is always a string (char *) and once there is no more parameters a NULL argument must be added at the end of the list.
The properties allowed per output plugin are specified on each specific plugin documentation.
Start Fluent Bit Engine
Once the library context has been created and the input/output plugin instances are set, the next step is to start the engine. When started, the engine runs inside a new thread (POSIX thread) without blocking the caller application. To start the engine the function flb_start() is used.
Prototype
Return Value
On success it returns 0; on error it returns a negative number.
Usage
This simple call only needs as argument ctx which is the reference to the context created at the beginning with flb_create():
Stop Fluent Bit Engine
To stop a running Fluent Bit engine, we provide the call flb_stop() for that purpose.
Prototype
The argument ctx is a reference to the context created at the beginning with flb_create() and previously started with flb_start().
When the call is invoked, the engine will wait a maximum of five seconds to flush buffers and release the resources in use. A stopped context can be re-started any time but without any data on it.
Return Value
On success it returns 0; on error it returns a negative number.
Usage
Destroy Library Context
A library context must be destroyed after is not longer necessary, note that a previous flb_stop() call is mandatory. When destroyed all resources associated are released.
Prototype
The argument ctx is a reference to the context created at the beginning with flb_create().
Return Value
No return value.
Usage
Ingest Data Manually
There are some cases where the caller application may want to ingest data into Fluent Bit, for this purpose exists the function flb_lib_push().
Prototype
The first argument is the context created previously through flb_create(). in_ffd is the numeric reference of the input plugin (for this case it should be an input of plugin lib type), data is a reference to the message to be ingested and len the number of bytes to take from it.
Return Value
On success, it returns the number of bytes written; on error it returns -1.
Usage
For more details and an example about how to use this function properly please refer to the next section .
Type
Description
flb_ctx_t
Main library context. It aims to reference the context returned by flb_create();
int ret;
ret = flb_service_set(ctx, "Flush", "1", NULL);
int flb_input(flb_ctx_t *ctx, char *name, void *data);
int in_ffd;
in_ffd = flb_input(ctx, "cpu", NULL);
int flb_input_set(flb_ctx_t *ctx, int in_ffd, ...);
int ret;
ret = flb_input_set(ctx, in_ffd,
"tag", "my_records",
"ssl", "false",
NULL);
int flb_output(flb_ctx_t *ctx, char *name, void *data);
int out_ffd;
out_ffd = flb_output(ctx, "stdout", NULL);
int flb_output_set(flb_ctx_t *ctx, int out_ffd, ...);
int ret;
ret = flb_output_set(ctx, out_ffd,
"tag", "my_records",
"ssl", "false",
NULL);
int flb_start(flb_ctx_t *ctx);
int ret;
ret = flb_start(ctx);
int flb_stop(flb_ctx_t *ctx);
int ret;
ret = flb_stop(ctx);
void flb_destroy(flb_ctx_t *ctx);
flb_destroy(ctx);
int flb_lib_push(flb_ctx_t *ctx, int in_ffd, void *data, size_t len);
Fluent Bit for Developers
Fluent Bit has been designed and built to be used not only as a standalone tool, it can also be embedded in your C or C++ applications. The following section presents details about how you can use it inside your own programs. We assume that you have some basic knowledge of C language, ideally experience compiling programs on Unix/Linux environments.
Ingest Records Manually
There are some cases where Fluent Bit library is used to send records from the caller application to some destination, this process is called manual data ingestion.
For this purpose a specific input plugin called lib exists and can be using in conjunction with the flb_lib_push() API function.
Data Format
The lib input plugin expect the data comes in a fixed JSON format as follows:
Every record must be a JSON array that contains at least two entries. The first one is the UNIX_TIMESTAMP which is a number representing time associated to the event generation (Epoch time) and the second entry is a JSON map with a list of key/values. A valid entry can be the following:
Usage
The following C code snippet shows how to insert a few JSON records into a running Fluent Bit engine:
Fluent Bit and Golang Plugins
Fluent Bit currently supports integration of Golang plugins built as shared objects for output plugins only. The interface for the Golang plugins is currently under development but is functional.
Getting Started
Compile Fluent Bit with Golang support, e.g:
$ cd build/
$ cmake -DFLB_DEBUG=On -DFLB_PROXY_GO=On ../
$ make
Once compiled, we can see a new option in the binary -e which stands for external plugin, e.g:
Build a Go Plugin
The fluent-bit-go package is available to assist developers in creating Go plugins.
At a minimum, a Go plugin looks like this:
the code above is a template to write an output plugin, it's really important to keep the package name as main and add an explicit main() function. This is a requirement as the code will be build as a shared library.
To build the code above, use the following line:
Once built, a shared library called out\_gstdout.so will be available. It's really important to double check the final .so file is what we expect. Doing a ldd over the library we should see something similar to this:
Run Fluent Bit with the new plugin
$ bin/fluent-bit -h
Usage: fluent-bit [OPTION]
Available Options
-c --config=FILE specify an optional configuration file
-d, --daemon run Fluent Bit in background mode
-f, --flush=SECONDS flush timeout in seconds (default: 5)
-i, --input=INPUT set an input
-m, --match=MATCH set plugin match, same as '-p match=abc'
-o, --output=OUTPUT set an output
-p, --prop="A=B" set plugin configuration property
-e, --plugin=FILE load an external plugin (shared lib)
...
#include <fluent-bit.h>
#define JSON_1 "[1449505010, {\"key1\": \"some value\"}]"
#define JSON_2 "[1449505620, {\"key1\": \"some new value\"}]"
int main()
{
int ret;
int in_ffd;
int out_ffd;
flb_ctx_t *ctx;
/* Create library context */
ctx = flb_create();
if (!ctx) {
return -1;
}
/* Enable the input plugin for manual data ingestion */
in_ffd = flb_input(ctx, "lib", NULL);
if (in_ffd == -1) {
flb_destroy(ctx);
return -1;
}
/* Enable output plugin 'stdout' (print records to the standard output) */
out_ffd = flb_output(ctx, "stdout", NULL);
if (out_ffd == -1) {
flb_destroy(ctx);
return -1;
}
/* Start the engine */
ret = flb_start(ctx);
if (ret == -1) {
flb_destroy(ctx);
return -1;
}
/* Ingest data manually */
flb_lib_push(ctx, in_ffd, JSON_1, sizeof(JSON_1) - 1);
flb_lib_push(ctx, in_ffd, JSON_2, sizeof(JSON_2) - 1);
/* Stop the engine (5 seconds to flush remaining data) */
flb_stop(ctx);
/* Destroy library context, release all resources */
flb_destroy(ctx);
return 0;
}
package main
import "github.com/fluent/fluent-bit-go/output"
//export FLBPluginRegister
func FLBPluginRegister(def unsafe.Pointer) int {
// Gets called only once when the plugin.so is loaded
return output.FLBPluginRegister(ctx, "gstdout", "Stdout GO!")
}
//export FLBPluginInit
func FLBPluginInit(plugin unsafe.Pointer) int {
// Gets called only once for each instance you have configured.
return output.FLB_OK
}
//export FLBPluginFlushCtx
func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int {
// Gets called with a batch of records to be written to an instance.
return output.FLB_OK
}
//export FLBPluginExit
func FLBPluginExit() int {
return output.FLB_OK
}
func main() {
}
$ go build -buildmode=c-shared -o out_gstdout.so out_gstdout.go