Fluent Bit has been designed and built to be used not only as a standalone tool, it can also be embedded in your C or C++ applications. The following section presents details about how you can use it inside your own programs. We assume that you have some basic knowledge of C language, ideally experience compiling programs on Unix/Linux environments.
There are some cases where Fluent Bit library is used to send records from the caller application to some destination, this process is called manual data ingestion.
For this purpose a specific input plugin called lib exists and can be using in conjunction with the flb_lib_push() API function.
The lib input plugin expect the data comes in a fixed JSON format as follows:
[UNIX_TIMESTAMP, MAP]Every record must be a JSON array that contains at least two entries. The first one is the UNIX_TIMESTAMP which is a number representing time associated to the event generation (Epoch time) and the second entry is a JSON map with a list of key/values. A valid entry can be the following:
The following C code snippet shows how to insert a few JSON records into a running Fluent Bit engine:
[1449505010, {"key1": "some value", "key2": false}]#include <fluent-bit.h>
#define JSON_1 "[1449505010, {\"key1\": \"some value\"}]"
#define JSON_2 "[1449505620, {\"key1\": \"some new value\"}]"
int main()
{
int ret;
int in_ffd;
int out_ffd;
flb_ctx_t *ctx;
/* Create library context */
ctx = flb_create();
if (!ctx) {
return -1;
}
/* Enable the input plugin for manual data ingestion */
in_ffd = flb_input(ctx, "lib", NULL);
if (in_ffd == -1) {
flb_destroy(ctx);
return -1;
}
/* Enable output plugin 'stdout' (print records to the standard output) */
out_ffd = flb_output(ctx, "stdout", NULL);
if (out_ffd == -1) {
flb_destroy(ctx);
return -1;
}
/* Start the engine */
ret = flb_start(ctx);
if (ret == -1) {
flb_destroy(ctx);
return -1;
}
/* Ingest data manually */
flb_lib_push(ctx, in_ffd, JSON_1, sizeof(JSON_1) - 1);
flb_lib_push(ctx, in_ffd, JSON_2, sizeof(JSON_2) - 1);
/* Stop the engine (5 seconds to flush remaining data) */
flb_stop(ctx);
/* Destroy library context, release all resources */
flb_destroy(ctx);
return 0;
}Fluent Bit currently supports integration of Golang plugins built as shared objects for output plugins only. The interface for the Golang plugins is currently under development but is functional.
Compile Fluent Bit with Golang support, e.g:
$ cd build/
$ cmake -DFLB_DEBUG=On -DFLB_PROXY_GO=On ../
$ makeOnce compiled, we can see a new option in the binary -e which stands for external plugin, e.g:
The fluent-bit-go package is available to assist developers in creating Go plugins.
At a minimum, a Go plugin looks like this:
the code above is a template to write an output plugin, it's really important to keep the package name as main and add an explicit main() function. This is a requirement as the code will be build as a shared library.
To build the code above, use the following line:
Once built, a shared library called out\_gstdout.so will be available. It's really important to double check the final .so file is what we expect. Doing a ldd over the library we should see something similar to this:
library is written in C language and can be used from any C or C++ application. Before digging into the specification it is recommended to understand the workflow involved in the runtime.
runs as a service, meaning that the API exposed for developers provide interfaces to create and manage a context, specify inputs/outputs, set configuration parameters and set routing paths for the event/records. A typical usage of the library involves:
$ bin/fluent-bit -h
Usage: fluent-bit [OPTION]
Available Options
-c --config=FILE specify an optional configuration file
-d, --daemon run Fluent Bit in background mode
-f, --flush=SECONDS flush timeout in seconds (default: 5)
-i, --input=INPUT set an input
-m, --match=MATCH set plugin match, same as '-p match=abc'
-o, --output=OUTPUT set an output
-p, --prop="A=B" set plugin configuration property
-e, --plugin=FILE load an external plugin (shared lib)
...$ go build -buildmode=c-shared -o out_gstdout.so out_gstdout.goEnable input plugin(s) and set properties.
Enable output plugin(s) and set properties.
Start the library runtime.
Optionally ingest records manually.
Stop the library runtime.
Destroy library instance/context.
Starting from Fluent Bit v0.9, there is only one data type exposed by the library, by convention prefixed with flb_.
Type
Description
flb_ctx_t
Main library context. It aims to reference the context returned by flb_create();
As described earlier, the first step to use the library is to create a context of it, for the purpose the function flb_create() is used.
Prototype
Return Value
On success, flb_create() returns the library context; on error, it returns NULL.
Usage
Using the flb_service_set() function is possible to set context properties.
Prototype
Return Value
On success it returns 0; on error it returns a negative number.
Usage
The flb_service_set() allows to set one or more properties in a key/value string mode, e.g:
The above example specified the values for the properties Flush , note that the value is always a string (char *) and once there is no more parameters a NULL argument must be added at the end of the list.
When built, Fluent Bit library contains a certain number of built-in input plugins. In order to enable an input plugin, the function flb_input() is used to create an instance of it.
For plugins, an instance means a context of the plugin enabled. You can create multiples instances of the same plugin.
Prototype
The argument ctx represents the library context created by flb_create(), then name is the name of the input plugin that is required to enable.
The third argument data can be used to pass a custom reference to the plugin instance, this is mostly used by custom or third party plugins, for generic plugins passing NULL is OK.
Return Value
On success, flb_input() returns an integer value >= zero (similar to a file descriptor); on error, it returns a negative number.
Usage
A plugin instance created through flb_input(), may provide some configuration properties. Using the flb_input_set() function is possible to set these properties.
Prototype
Return Value
On success it returns 0; on error it returns a negative number.
Usage
The flb_input_set() allows to set one or more properties in a key/value string mode, e.g:
The argument ctx represents the library context created by flb_create(). The above example specified the values for the properties tag and ssl, note that the value is always a string (char *) and once there is no more parameters a NULL argument must be added at the end of the list.
The properties allowed per input plugin are specified on each specific plugin documentation.
When built, Fluent Bit library contains a certain number of built-in output plugins. In order to enable an output plugin, the function flb_output() is used to create an instance of it.
For plugins, an instance means a context of the plugin enabled. You can create multiples instances of the same plugin.
Prototype
The argument ctx represents the library context created by flb_create(), then name is the name of the output plugin that is required to enable.
The third argument data can be used to pass a custom reference to the plugin instance, this is mostly used by custom or third party plugins, for generic plugins passing NULL is OK.
Return Value
On success, flb_output() returns the output plugin instance; on error, it returns a negative number.
Usage
A plugin instance created through flb_output(), may provide some configuration properties. Using the flb_output_set() function is possible to set these properties.
Prototype
Return Value
On success it returns an integer value >= zero (similar to a file descriptor); on error it returns a negative number.
Usage
The flb_output_set() allows to set one or more properties in a key/value string mode, e.g:
The argument ctx represents the library context created by flb_create(). The above example specified the values for the properties tag and ssl, note that the value is always a string (char *) and once there is no more parameters a NULL argument must be added at the end of the list.
The properties allowed per output plugin are specified on each specific plugin documentation.
Once the library context has been created and the input/output plugin instances are set, the next step is to start the engine. When started, the engine runs inside a new thread (POSIX thread) without blocking the caller application. To start the engine the function flb_start() is used.
Prototype
Return Value
On success it returns 0; on error it returns a negative number.
Usage
This simple call only needs as argument ctx which is the reference to the context created at the beginning with flb_create():
To stop a running Fluent Bit engine, we provide the call flb_stop() for that purpose.
Prototype
The argument ctx is a reference to the context created at the beginning with flb_create() and previously started with flb_start().
When the call is invoked, the engine will wait a maximum of five seconds to flush buffers and release the resources in use. A stopped context can be re-started any time but without any data on it.
Return Value
On success it returns 0; on error it returns a negative number.
Usage
A library context must be destroyed after is not longer necessary, note that a previous flb_stop() call is mandatory. When destroyed all resources associated are released.
Prototype
The argument ctx is a reference to the context created at the beginning with flb_create().
Return Value
No return value.
Usage
There are some cases where the caller application may want to ingest data into Fluent Bit, for this purpose exists the function flb_lib_push().
Prototype
The first argument is the context created previously through flb_create(). in_ffd is the numeric reference of the input plugin (for this case it should be an input of plugin lib type), data is a reference to the message to be ingested and len the number of bytes to take from it.
Return Value
On success, it returns the number of bytes written; on error it returns -1.
Usage
For more details and an example about how to use this function properly please refer to the next section Ingest Records Manually.
package main
import "github.com/fluent/fluent-bit-go/output"
//export FLBPluginRegister
func FLBPluginRegister(def unsafe.Pointer) int {
// Gets called only once when the plugin.so is loaded
return output.FLBPluginRegister(ctx, "gstdout", "Stdout GO!")
}
//export FLBPluginInit
func FLBPluginInit(plugin unsafe.Pointer) int {
// Gets called only once for each instance you have configured.
return output.FLB_OK
}
//export FLBPluginFlushCtx
func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int {
// Gets called with a batch of records to be written to an instance.
return output.FLB_OK
}
//export FLBPluginExit
func FLBPluginExit() int {
return output.FLB_OK
}
func main() {
}$ ldd out_gstdout.so
linux-vdso.so.1 => (0x00007fff561dd000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc4aeef0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc4aeb27000)
/lib64/ld-linux-x86-64.so.2 (0x000055751a4fd000)$ bin/fluent-bit -e /path/to/out_gstdout.so -i cpu -o gstdoutflb_ctx_t *flb_create();flb_ctx_t *ctx;
ctx = flb_create();
if (!ctx) {
return NULL;
}int flb_service_set(flb_ctx_t *ctx, ...);int ret;
ret = flb_service_set(ctx, "Flush", "1", NULL);int flb_input(flb_ctx_t *ctx, char *name, void *data);int in_ffd;
in_ffd = flb_input(ctx, "cpu", NULL);int flb_input_set(flb_ctx_t *ctx, int in_ffd, ...);int ret;
ret = flb_input_set(ctx, in_ffd,
"tag", "my_records",
"ssl", "false",
NULL);int flb_output(flb_ctx_t *ctx, char *name, void *data);int out_ffd;
out_ffd = flb_output(ctx, "stdout", NULL);int flb_output_set(flb_ctx_t *ctx, int out_ffd, ...);int ret;
ret = flb_output_set(ctx, out_ffd,
"tag", "my_records",
"ssl", "false",
NULL);int flb_start(flb_ctx_t *ctx);int ret;
ret = flb_start(ctx);int flb_stop(flb_ctx_t *ctx);int ret;
ret = flb_stop(ctx);void flb_destroy(flb_ctx_t *ctx);flb_destroy(ctx);int flb_lib_push(flb_ctx_t *ctx, int in_ffd, void *data, size_t len);