1 of 27

Inputs

Collectd

The collectd input plugin allows you to receive datagrams from collectd service.

Configuration Parameters

The plugin supports the following configuration parameters:

Key

Description

Default

Listen

Set the address to listen to

0.0.0.0

Port

Set the port to listen to

25826

TypesDB

Set the data specification file

/usr/share/collectd/types.db

Configuration Examples

Here is a basic configuration example.

[INPUT]
    Name         collectd
    Listen       0.0.0.0
    Port         25826
    TypesDB      /usr/share/collectd/types.db,/etc/collectd/custom.db

[OUTPUT]
    Name   stdout
    Match  *

With this configuration, Fluent Bit listens to 0.0.0.0:25826, and outputs incoming datagram packets to stdout.

You must set the same types.db files that your collectd server uses. Otherwise, Fluent Bit may not be able to interpret the payload properly.

CPU Metrics

The cpu input plugin, measures the CPU usage of a process or the whole system by default (considering per CPU core). It reports values in percentage unit for every interval of time set. At the moment this plugin is only available for Linux.

The following tables describes the information generated by the plugin. The keys below represent the data used by the overall system, all values associated to the keys are in a percentage unit (0 to 100%):

key

description

cpu_p

CPU usage of the overall system, this value is the summatory of time spent on user and kernel space. The result takes in consideration the numbers of CPU cores in the system.

user_p

CPU usage in User mode, for short it means the CPU usage by user space programs. The result of this value takes in consideration the numbers of CPU cores in the system.

system_p

CPU usage in Kernel mode, for short it means the CPU usage by the Kernel. The result of this value takes in consideration the numbers of CPU cores in the system.

In addition to the keys reported in the above table, a similar content is created per CPU core. The cores are listed from 0 to N as the Kernel reports:

key

description

cpuN.p_cpu

Represents the total CPU usage by core N.

cpuN.p_user

Total CPU spent in user mode or user space programs associated to this core.

cpuN.p_system

Total CPU spent in system or kernel mode associated to this core.

Configuration Parameters

The plugin supports the following configuration parameters:

Key

Description

Default

Interval_Sec

Polling interval in seconds

Interval_NSec

Polling interval in nanoseconds

PID

Specify the ID (PID) of a running process in the system. By default the plugin monitors the whole system but if this option is set, it will only monitor the given process ID.

Getting Started

In order to get the statistics of the CPU usage of your system, you can run the plugin from the command line or through the configuration file:

Command Line

$ build/bin/fluent-bit -i cpu -t my_cpu -o stdout -m '*'
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2019/09/02 10:46:29] [ info] starting engine
[0] [1452185189, {"cpu_p"=>7.00, "user_p"=>5.00, "system_p"=>2.00, "cpu0.p_cpu"=>10.00, "cpu0.p_user"=>8.00, "cpu0.p_system"=>2.00, "cpu1.p_cpu"=>6.00, "cpu1.p_user"=>4.00, "cpu1.p_system"=>2.00}]
[1] [1452185190, {"cpu_p"=>6.50, "user_p"=>5.00, "system_p"=>1.50, "cpu0.p_cpu"=>6.00, "cpu0.p_user"=>5.00, "cpu0.p_system"=>1.00, "cpu1.p_cpu"=>7.00, "cpu1.p_user"=>5.00, "cpu1.p_system"=>2.00}]
[2] [1452185191, {"cpu_p"=>7.50, "user_p"=>5.00, "system_p"=>2.50, "cpu0.p_cpu"=>7.00, "cpu0.p_user"=>3.00, "cpu0.p_system"=>4.00, "cpu1.p_cpu"=>6.00, "cpu1.p_user"=>6.00, "cpu1.p_system"=>0.00}]
[3] [1452185192, {"cpu_p"=>4.50, "user_p"=>3.50, "system_p"=>1.00, "cpu0.p_cpu"=>6.00, "cpu0.p_user"=>5.00, "cpu0.p_system"=>1.00, "cpu1.p_cpu"=>5.00, "cpu1.p_user"=>3.00, "cpu1.p_system"=>2.00}]

As described above, the CPU input plugin gathers the overall usage every one second and flushed the information to the output on the fifth second. On this example we used the stdout plugin to demonstrate the output records. In a real use-case you may want to flush this information to some central aggregator such as Fluentd or Elasticsearch.

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name cpu
    Tag  my_cpu

[OUTPUT]
    Name  stdout
    Match *

Dummy

The dummy input plugin, generates dummy events. It is useful for testing, debugging, benchmarking and getting started with Fluent Bit.

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

You can run the plugin from the command line or through the configuration file:

Key

Description

Dummy

Dummy JSON record. Default: {"message":"dummy"}

Start_time_sec

Dummy base timestamp in seconds. Default: 0

Start_time_nsec

Dummy base timestamp in nanoseconds. Default: 0

Rate

Events number generated per second. Default: 1

Samples

If set, the events number will be limited. e.g. If Samples=3, the plugin only generates three events and stops.

Command Line

$ fluent-bit -i dummy -o stdout
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2017/07/06 21:55:29] [ info] [engine] started
[0] dummy.0: [1499345730.015265366, {"message"=>"dummy"}]
[1] dummy.0: [1499345731.002371371, {"message"=>"dummy"}]
[2] dummy.0: [1499345732.000267932, {"message"=>"dummy"}]
[3] dummy.0: [1499345733.000757746, {"message"=>"dummy"}]

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name   dummy
    Tag    dummy.log

[OUTPUT]
    Name   stdout
    Match  *

Exec

The exec input plugin, allows to execute external program and collects event logs.

Configuration Parameters

The plugin supports the following configuration parameters:

Key

Description

Command

The command to execute.

Parser

Specify the name of a parser to interpret the entry as a structured message.

Interval_Sec

Polling interval (seconds).

Interval_NSec

Polling interval (nanosecond).

Buf_Size

Getting Started

You can run the plugin from the command line or through the configuration file:

Command Line

The following example will read events from the output of ls.

$ fluent-bit -i exec -p 'command=ls /var/log' -o stdout
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2018/03/21 17:46:49] [ info] [engine] started
[0] exec.0: [1521622010.013470159, {"exec"=>"ConsoleKit"}]
[1] exec.0: [1521622010.013490313, {"exec"=>"Xorg.0.log"}]
[2] exec.0: [1521622010.013492079, {"exec"=>"Xorg.0.log.old"}]
[3] exec.0: [1521622010.013493443, {"exec"=>"anaconda.ifcfg.log"}]
[4] exec.0: [1521622010.013494707, {"exec"=>"anaconda.log"}]
[5] exec.0: [1521622010.013496016, {"exec"=>"anaconda.program.log"}]
[6] exec.0: [1521622010.013497225, {"exec"=>"anaconda.storage.log"}]

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name          exec
    Tag           exec_ls
    Command       ls /var/log
    Interval_Sec  1
    Interval_NSec 0
    Buf_Size      8mb

[OUTPUT]
    Name   stdout
    Match  *

Head

The head input plugin, allows to read events from the head of file. It's behavior is similar to the head command.

Configuration Parameters

The plugin supports the following configuration parameters:

Key

Description

File

Absolute path to the target file, e.g: /proc/uptime

Buf_Size

Buffer size to read the file.

Interval_Sec

Polling interval (seconds).

Interval_NSec

Polling interval (nanosecond).

Add_Path

If enabled, filepath is appended to each records. Default value is false.

Key

Rename a key. Default: head.

Lines

Line number to read. If the number N is set, in_head reads first N lines like head(1) -n.

Split_line

If enabled, in_head generates key-value pair per line.

Split Line Mode

This mode is useful to get a specific line. This is an example to get CPU frequency from /proc/cpuinfo.

/proc/cpuinfo is a special file to get cpu information.

processor    : 0
vendor_id    : GenuineIntel
cpu family   : 6
model        : 42
model name   : Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
stepping     : 7
microcode    : 41
cpu MHz      : 2791.009
cache size   : 4096 KB
physical id  : 0
siblings     : 1

Cpu frequency is "cpu MHz : 2791.009". We can get the line with this configuration file.

[INPUT]
    Name           head
    Tag            head.cpu
    File           /proc/cpuinfo
    Lines          8
    Split_line     true
    # {"line0":"processor    : 0", "line1":"vendor_id    : GenuineIntel" ...}

[FILTER]
    Name           record_modifier
    Match          *
    Whitelist_key  line7

[OUTPUT]
    Name           stdout
    Match          *

Output is

$ bin/fluent-bit -c head.conf 
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2017/06/26 22:38:24] [ info] [engine] started
[0] head.cpu: [1498484305.000279805, {"line7"=>"cpu MHz        : 2791.009"}]
[1] head.cpu: [1498484306.011680137, {"line7"=>"cpu MHz        : 2791.009"}]
[2] head.cpu: [1498484307.010042482, {"line7"=>"cpu MHz        : 2791.009"}]
[3] head.cpu: [1498484308.008447978, {"line7"=>"cpu MHz        : 2791.009"}]

Getting Started

In order to read the head of a file, you can run the plugin from the command line or through the configuration file:

Command Line

The following example will read events from the /proc/uptime file, tag the records with the uptime name and flush them back to the stdout plugin:

$ fluent-bit -i head -t uptime -p File=/proc/uptime -o stdout -m '*'
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2016/05/17 21:53:54] [ info] starting engine
[0] uptime: [1463543634, {"head"=>"133517.70 194870.97"}]
[1] uptime: [1463543635, {"head"=>"133518.70 194872.85"}]
[2] uptime: [1463543636, {"head"=>"133519.70 194876.63"}]
[3] uptime: [1463543637, {"head"=>"133520.70 194879.72"}]

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name          head
    Tag           uptime
    File          /proc/uptime
    Buf_Size      256
    Interval_Sec  1
    Interval_NSec 0

[OUTPUT]
    Name   stdout
    Match  *

Note: Total interval (sec) = Interval_Sec + (Interval_Nsec / 1000000000).

e.g. 1.5s = 1s + 500000000ns

HTTP

The HTTP input plugin allows you to send custom records to an HTTP endpoint.

Configuration Parameters

Key

Description

default

host

The address to listen on

0.0.0.0

port

The port for Fluent Bit to listen on

9880

buffer_max_size

Specify the maximum buffer size in KB to receive a JSON message.

buffer_chunk_size

This sets the chunk size for incoming incoming JSON messages. These chunks are then stored/managed in the space available by buffer_max_size.

512K

Getting Started

The http input plugin allows Fluent Bit to open up an HTTP port that you can then route data to in a dynamic way. This plugin supports dynamic tags which allow you to send data with different tags through the same input. An example video and curl message can be seen below

Link to video

How to set tag

The tag for the HTTP input plugin is set by adding the tag to the end of the request URL. This tag is then used to route the event through the system. For example, in the following curl message below the tag set is app.log. If you do not set the tag http.0 is automatically used. If you have multiple HTTP inputs then they will follow a pattern of http.N where N is an integer representing the input.

Example Curl message

curl -d @app.log -XPOST -H "content-type: application/json" http://localhost:8888/app.log

Configuration File

[INPUT]
    name http
    host 0.0.0.0
    port 8888

[OUTPUT]
    type stdout
    match *

Command Line

$> fluent-bit -i http -p port=8888 -o stdout

CPU Metrics

key

description

cpu_p

CPU usage of the overall system, this value is the summatory of time spent on user and kernel space. The result takes in consideration the numbers of CPU cores in the system.

user_p

CPU usage in User mode, for short it means the CPU usage by user space programs. The result of this value takes in consideration the numbers of CPU cores in the system.

system_p

CPU usage in Kernel mode, for short it means the CPU usage by the Kernel. The result of this value takes in consideration the numbers of CPU cores in the system.

In addition to the keys reported in the above table, a similar content is created per CPU core. The cores are listed from 0 to N as the Kernel reports:

key

description

cpuN.p_cpu

Represents the total CPU usage by core N.

cpuN.p_user

Total CPU spent in user mode or user space programs associated to this core.

cpuN.p_system

Total CPU spent in system or kernel mode associated to this core.

Configuration Parameters

The plugin supports the following configuration parameters:

Key

Description

Default

Interval_Sec

Polling interval in seconds

Interval_NSec

Polling interval in nanoseconds

PID

Specify the ID (PID) of a running process in the system. By default the plugin monitors the whole system but if this option is set, it will only monitor the given process ID.

Getting Started

In order to get the statistics of the CPU usage of your system, you can run the plugin from the command line or through the configuration file:

Command Line

$ build/bin/fluent-bit -i cpu -t my_cpu -o stdout -m '*'
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2019/09/02 10:46:29] [ info] starting engine
[0] [1452185189, {"cpu_p"=>7.00, "user_p"=>5.00, "system_p"=>2.00, "cpu0.p_cpu"=>10.00, "cpu0.p_user"=>8.00, "cpu0.p_system"=>2.00, "cpu1.p_cpu"=>6.00, "cpu1.p_user"=>4.00, "cpu1.p_system"=>2.00}]
[1] [1452185190, {"cpu_p"=>6.50, "user_p"=>5.00, "system_p"=>1.50, "cpu0.p_cpu"=>6.00, "cpu0.p_user"=>5.00, "cpu0.p_system"=>1.00, "cpu1.p_cpu"=>7.00, "cpu1.p_user"=>5.00, "cpu1.p_system"=>2.00}]
[2] [1452185191, {"cpu_p"=>7.50, "user_p"=>5.00, "system_p"=>2.50, "cpu0.p_cpu"=>7.00, "cpu0.p_user"=>3.00, "cpu0.p_system"=>4.00, "cpu1.p_cpu"=>6.00, "cpu1.p_user"=>6.00, "cpu1.p_system"=>0.00}]
[3] [1452185192, {"cpu_p"=>4.50, "user_p"=>3.50, "system_p"=>1.00, "cpu0.p_cpu"=>6.00, "cpu0.p_user"=>5.00, "cpu0.p_system"=>1.00, "cpu1.p_cpu"=>5.00, "cpu1.p_user"=>3.00, "cpu1.p_system"=>2.00}]

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name cpu
    Tag  my_cpu

[OUTPUT]
    Name  stdout
    Match *

Head

The head input plugin, allows to read events from the head of file. It's behavior is similar to the head command.

Configuration Parameters

The plugin supports the following configuration parameters:

Key

Description

File

Absolute path to the target file, e.g: /proc/uptime

Buf_Size

Buffer size to read the file.

Interval_Sec

Polling interval (seconds).

Interval_NSec

Polling interval (nanosecond).

Add_Path

If enabled, filepath is appended to each records. Default value is false.

Key

Rename a key. Default: head.

Lines

Line number to read. If the number N is set, in_head reads first N lines like head(1) -n.

Split_line

If enabled, in_head generates key-value pair per line.

Split Line Mode

This mode is useful to get a specific line. This is an example to get CPU frequency from /proc/cpuinfo.

/proc/cpuinfo is a special file to get cpu information.

processor    : 0
vendor_id    : GenuineIntel
cpu family   : 6
model        : 42
model name   : Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
stepping     : 7
microcode    : 41
cpu MHz      : 2791.009
cache size   : 4096 KB
physical id  : 0
siblings     : 1

Cpu frequency is "cpu MHz : 2791.009". We can get the line with this configuration file.

[INPUT]
    Name           head
    Tag            head.cpu
    File           /proc/cpuinfo
    Lines          8
    Split_line     true
    # {"line0":"processor    : 0", "line1":"vendor_id    : GenuineIntel" ...}

[FILTER]
    Name           record_modifier
    Match          *
    Whitelist_key  line7

[OUTPUT]
    Name           stdout
    Match          *

Output is

$ bin/fluent-bit -c head.conf 
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2017/06/26 22:38:24] [ info] [engine] started
[0] head.cpu: [1498484305.000279805, {"line7"=>"cpu MHz        : 2791.009"}]
[1] head.cpu: [1498484306.011680137, {"line7"=>"cpu MHz        : 2791.009"}]
[2] head.cpu: [1498484307.010042482, {"line7"=>"cpu MHz        : 2791.009"}]
[3] head.cpu: [1498484308.008447978, {"line7"=>"cpu MHz        : 2791.009"}]

Getting Started

In order to read the head of a file, you can run the plugin from the command line or through the configuration file:

Command Line

The following example will read events from the /proc/uptime file, tag the records with the uptime name and flush them back to the stdout plugin:

$ fluent-bit -i head -t uptime -p File=/proc/uptime -o stdout -m '*'
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2016/05/17 21:53:54] [ info] starting engine
[0] uptime: [1463543634, {"head"=>"133517.70 194870.97"}]
[1] uptime: [1463543635, {"head"=>"133518.70 194872.85"}]
[2] uptime: [1463543636, {"head"=>"133519.70 194876.63"}]
[3] uptime: [1463543637, {"head"=>"133520.70 194879.72"}]

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name          head
    Tag           uptime
    File          /proc/uptime
    Buf_Size      256
    Interval_Sec  1
    Interval_NSec 0

[OUTPUT]
    Name   stdout
    Match  *

Note: Total interval (sec) = Interval_Sec + (Interval_Nsec / 1000000000).

e.g. 1.5s = 1s + 500000000ns