The collectd input plugin allows you to receive datagrams from collectd service.

Configuration Parameters

The plugin supports the following configuration parameters:

Configuration Examples

Here is a basic configuration example.

[INPUT]
    Name         collectd
    Listen       0.0.0.0
    Port         25826
    TypesDB      /usr/share/collectd/types.db,/etc/collectd/custom.db

[OUTPUT]
    Name   stdout
    Match  *

With this configuration, Fluent Bit listens to 0.0.0.0:25826, and outputs incoming datagram packets to stdout.

You must set the same types.db files that your collectd server uses. Otherwise, Fluent Bit may not be able to interpret the payload properly.

The docker events input plugin uses the docker API to capture server events. A complete list of possible events returned by this plugin can be found

Configuration Parameters

This plugin supports the following configuration parameters:

Command Line

Configuration File

In your main configuration file append the following Input & Output sections:

The elasticsearch input plugin handles both Elasticsearch and OpenSearch Bulk API requests.

Configuration Parameters

The plugin supports the following configuration parameters:

Note: The Elasticsearch cluster uses "sniffing" to optimize the connections between its cluster and clients. Elasticsearch can build its cluster and dynamically generate a connection list which is called "sniffing". The hostname will be used for sniffing information and this is handled by the sniffing endpoint.

Getting Started

In order to start performing the checks, you can run the plugin from the command line or through the configuration file:

Command Line

From the command line you can configure Fluent Bit to handle Bulk API requests with the following options:

Configuration File

In your main configuration file append the following Input & Output sections:

As described above, the plugin will handle ingested Bulk API requests. For large bulk ingestions, you may have to increase buffer size with buffer_max_size and buffer_chunk_size parameters:

Ingesting from beats series

Note that Fluent Bit's node information is returning as Elasticsearch 8.0.0.

So, users have to specify the following configurations on their beats configurations:

For large log ingestion on these beat plugins, users might have to configure rate limiting on those beats plugins when Fluent Bit indicates that the application is exceeding the size limit for HTTP requests:

The cpu input plugin, measures the CPU usage of a process or the whole system by default (considering per CPU core). It reports values in percentage unit for every interval of time set. At the moment this plugin is only available for Linux.

The following tables describes the information generated by the plugin. The keys below represent the data used by the overall system, all values associated to the keys are in a percentage unit (0 to 100%):

The CPU metrics plugin creates metrics that are log-based, such as JSON payload. For Prometheus-based metrics, see the Node Exporter Metrics input plugin.

In addition to the keys reported in the above table, a similar content is created per CPU core. The cores are listed from 0 to N as the Kernel reports:

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

In order to get the statistics of the CPU usage of your system, you can run the plugin from the command line or through the configuration file:

Command Line

Configuration File

In your main configuration file append the following Input & Output sections:

The disk input plugin, gathers the information about the disk throughput of the running system every certain interval of time and reports them.

The Disk I/O metrics plugin creates metrics that are log-based, such as JSON payload. For Prometheus-based metrics, see the Node Exporter Metrics input plugin.

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

In order to get disk usage from your system, you can run the plugin from the command line or through the configuration file:

Command Line

$ fluent-bit -i disk -o stdout
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2017/01/28 16:58:16] [ info] [engine] started
[0] disk.0: [1485590297, {"read_size"=>0, "write_size"=>0}]
[1] disk.0: [1485590298, {"read_size"=>0, "write_size"=>0}]
[2] disk.0: [1485590299, {"read_size"=>0, "write_size"=>0}]
[3] disk.0: [1485590300, {"read_size"=>0, "write_size"=>11997184}]

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name          disk
    Tag           disk
    Interval_Sec  1
    Interval_NSec 0
[OUTPUT]
    Name   stdout
    Match  *

pipeline:
    inputs:
        - name: disk
          tag: disk
          interval_sec: 1
          interval_nsec: 0
    outputs:
        - name: stdout
          match: '*'

Note: Total interval (sec) = Interval_Sec + (Interval_Nsec / 1000000000).

e.g. 1.5s = 1s + 500000000ns

The head input plugin, allows to read events from the head of file. It's behavior is similar to the head command.

Configuration Parameters

The plugin supports the following configuration parameters:

Split Line Mode

This mode is useful to get a specific line. This is an example to get CPU frequency from /proc/cpuinfo.

/proc/cpuinfo is a special file to get cpu information.

processor    : 0
vendor_id    : GenuineIntel
cpu family   : 6
model        : 42
model name   : Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
stepping     : 7
microcode    : 41
cpu MHz      : 2791.009
cache size   : 4096 KB
physical id  : 0
siblings     : 1

Cpu frequency is "cpu MHz : 2791.009". We can get the line with this configuration file.

[INPUT]
    Name           head
    Tag            head.cpu
    File           /proc/cpuinfo
    Lines          8
    Split_line     true
    # {"line0":"processor    : 0", "line1":"vendor_id    : GenuineIntel" ...}

[FILTER]
    Name           record_modifier
    Match          *
    Whitelist_key  line7

[OUTPUT]
    Name           stdout
    Match          *

pipeline:
    inputs:
        - name: head
          tag: head.cpu
          file: /proc/cpuinfo
          lines: 8
          split_line: true
    filters:
        - name: record_modifier
          match: '*'
          whitelist_key: line7
    outputs:
        - name: stdout
          match: '*'

Output is

$ bin/fluent-bit -c head.conf
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2017/06/26 22:38:24] [ info] [engine] started
[0] head.cpu: [1498484305.000279805, {"line7"=>"cpu MHz        : 2791.009"}]
[1] head.cpu: [1498484306.011680137, {"line7"=>"cpu MHz        : 2791.009"}]
[2] head.cpu: [1498484307.010042482, {"line7"=>"cpu MHz        : 2791.009"}]
[3] head.cpu: [1498484308.008447978, {"line7"=>"cpu MHz        : 2791.009"}]

Getting Started

In order to read the head of a file, you can run the plugin from the command line or through the configuration file:

Command Line

The following example will read events from the /proc/uptime file, tag the records with the uptime name and flush them back to the stdout plugin:

$ fluent-bit -i head -t uptime -p File=/proc/uptime -o stdout -m '*'
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2016/05/17 21:53:54] [ info] starting engine
[0] uptime: [1463543634, {"head"=>"133517.70 194870.97"}]
[1] uptime: [1463543635, {"head"=>"133518.70 194872.85"}]
[2] uptime: [1463543636, {"head"=>"133519.70 194876.63"}]
[3] uptime: [1463543637, {"head"=>"133520.70 194879.72"}]

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name          head
    Tag           uptime
    File          /proc/uptime
    Buf_Size      256
    Interval_Sec  1
    Interval_NSec 0

[OUTPUT]
    Name   stdout
    Match  *

pipeline:
    inputs:
        - name: head
          tag: uptime
          file: /proc/uptime
          buf_size: 256
          interval_sec: 1
          interval_nsec: 0
    outputs:
        - name: stdout
          match: '*'

Note: Total interval (sec) = Interval_Sec + (Interval_Nsec / 1000000000).

e.g. 1.5s = 1s + 500000000ns

The exec input plugin, allows to execute external program and collects event logs.

WARNING: Because this plugin invokes commands via a shell, its inputs are subject to shell metacharacter substitution. Careless use of untrusted input in command arguments could lead to malicious command execution.

Container support

This plugin will not function in all the distroless production images as it needs a functional /bin/sh which is not present. The debug images use the same binaries so even though they have a shell, there is no support for this plugin as it is compiled out.

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

You can run the plugin from the command line or through the configuration file:

Command Line

The following example will read events from the output of ls.

$ fluent-bit -i exec -p 'command=ls /var/log' -o stdout
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2018/03/21 17:46:49] [ info] [engine] started
[0] exec.0: [1521622010.013470159, {"exec"=>"ConsoleKit"}]
[1] exec.0: [1521622010.013490313, {"exec"=>"Xorg.0.log"}]
[2] exec.0: [1521622010.013492079, {"exec"=>"Xorg.0.log.old"}]
[3] exec.0: [1521622010.013493443, {"exec"=>"anaconda.ifcfg.log"}]
[4] exec.0: [1521622010.013494707, {"exec"=>"anaconda.log"}]
[5] exec.0: [1521622010.013496016, {"exec"=>"anaconda.program.log"}]
[6] exec.0: [1521622010.013497225, {"exec"=>"anaconda.storage.log"}]

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name          exec
    Tag           exec_ls
    Command       ls /var/log
    Interval_Sec  1
    Interval_NSec 0
    Buf_Size      8mb
    Oneshot       false

[OUTPUT]
    Name   stdout
    Match  *

pipeline:
    inputs:
        - name: exec
          tag: exec_ls
          command: ls /var/log
          interval_sec: 1
          interval_nsec: 0
          buf_size: 8mb
          oneshot: false

    outputs:
        - name: stdout
          match: '*'

Use as a command wrapper

To use fluent-bit with the exec plugin to wrap another command, use the Exit_After_Oneshot and Propagate_Exit_Code options, e.g.:

[INPUT]
    Name                exec
    Tag                 exec_oneshot_demo
    Command             for s in $(seq 1 10); do echo "count: $s"; sleep 1; done; exit 1
    Oneshot             true
    Exit_After_Oneshot  true
    Propagate_Exit_Code true

[OUTPUT]
    Name   stdout
    Match  *

pipeline:
    inputs:
        - name: exec
          tag: exec_oneshot_demo
          command: 'for s in $(seq 1 10); do echo "count: $s"; sleep 1; done; exit 1'
          oneshot: true
          exit_after_oneshot: true
          propagate_exit_code: true

    outputs:
        - name: stdout
          match: '*'

fluent-bit will output

[0] exec_oneshot_demo: [[1681702172.950574027, {}], {"exec"=>"count: 1"}]
[1] exec_oneshot_demo: [[1681702173.951663666, {}], {"exec"=>"count: 2"}]
[2] exec_oneshot_demo: [[1681702174.953873724, {}], {"exec"=>"count: 3"}]
[3] exec_oneshot_demo: [[1681702175.955760865, {}], {"exec"=>"count: 4"}]
[4] exec_oneshot_demo: [[1681702176.956840282, {}], {"exec"=>"count: 5"}]
[5] exec_oneshot_demo: [[1681702177.958292246, {}], {"exec"=>"count: 6"}]
[6] exec_oneshot_demo: [[1681702178.959508200, {}], {"exec"=>"count: 7"}]
[7] exec_oneshot_demo: [[1681702179.961715745, {}], {"exec"=>"count: 8"}]
[8] exec_oneshot_demo: [[1681702180.963924140, {}], {"exec"=>"count: 9"}]
[9] exec_oneshot_demo: [[1681702181.965852990, {}], {"exec"=>"count: 10"}]

then exit with exit code 1.

Translation of command exit code(s) to fluent-bit exit code follows the usual shell rules for exit code handling. Like with a shell, there is no way to differentiate between the command exiting on a signal and the shell exiting on a signal, and no way to differentiate between normal exits with codes greater than 125 and abnormal or signal exits reported by fluent-bit or the shell. Wrapped commands should use exit codes between 0 and 125 inclusive to allow reliable identification of normal exit. If the command is a pipeline, the exit code will be the exit code of the last command in the pipeline unless overridden by shell options.

Parsing command output

By default the exec plugin emits one message per command output line, with a single field exec containing the full message. Use the Parser directive to specify the name of a parser configuration to use to process the command input.

Security concerns

Take great care with shell quoting and escaping when wrapping commands. A script like

#!/bin/bash
# This is a DANGEROUS example of what NOT to do, NEVER DO THIS
exec fluent-bit \
  -o stdout \
  -i exec \
  -p exit_after_oneshot=true \
  -p propagate_exit_code=true \
  -p command='myscript $*'

can ruin your day if someone passes it the argument $(rm -rf /my/important/files; echo "deleted your stuff!")'

The above script would be safer if written with:

  -p command='echo '"$(printf '%q' "$@")" \

... but it's generally best to avoid dynamically generating the command or handling untrusted arguments to it at all.

The dummy input plugin, generates dummy events. It is useful for testing, debugging, benchmarking and getting started with Fluent Bit.

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

You can run the plugin from the command line or through the configuration file:

Command Line

$ fluent-bit -i dummy -o stdout
Fluent Bit v2.x.x
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[0] dummy.0: [[1686451466.659962491, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1686451467.659679509, {}], {"message"=>"dummy"}]

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name   dummy
    Dummy {"message": "custom dummy"}

[OUTPUT]
    Name   stdout
    Match  *

pipeline:
  inputs:
    - name: dummy
      dummy: '{"message": "custom dummy"}'
  outputs:
    - name: stdout
      match: '*'

Configuration Parameters

The plugin supports the following configuration parameters:

If you set neither Include nor Exclude, the plugin will try to get metrics from all the running containers.

Configuration File

Here is an example configuration that collects metrics from two docker instances (6bab19c3a0f9 and 14159be4ca2c).

[INPUT]
    Name         docker
    Include      6bab19c3a0f9 14159be4ca2c
[OUTPUT]
    Name   stdout
    Match  *

pipeline:
    inputs:
        - name: docker
          include: 6bab19c3a0f9 14159be4ca2c

    outputs:
        - name: stdout
          match: '*'

This configuration will produce records like below.

[1] docker.0: [1571994772.00555745, {"id"=>"6bab19c3a0f9", "name"=>"postgresql", "cpu_used"=>172102435, "mem_used"=>5693400, "mem_limit"=>4294963200}]

Fluent Bit exposes its own metrics to allow you to monitor the internals of your pipeline. The collected metrics can be processed similarly to those from the Prometheus Node Exporter input plugin. They can be sent to output plugins including Prometheus Exporter, Prometheus Remote Write or OpenTelemetry..

Important note: Metrics collected with Node Exporter Metrics flow through a separate pipeline from logs and current filters do not operate on top of metrics.

Configuration

Getting Started

Simple Configuration File

In the following configuration file, the input plugin _node_exporter_metrics collects _metrics every 2 seconds and exposes them through our Prometheus Exporter output plugin on HTTP/TCP port 2021.

# Fluent Bit Metrics + Prometheus Exporter
# -------------------------------------------
# The following example collects Fluent Bit metrics and exposes
# them through a Prometheus HTTP end-point.
#
# After starting the service try it with:
#
# $ curl http://127.0.0.1:2021/metrics
#
[SERVICE]
    flush           1
    log_level       info

[INPUT]
    name            fluentbit_metrics
    tag             internal_metrics
    scrape_interval 2

[OUTPUT]
    name            prometheus_exporter
    match           internal_metrics
    host            0.0.0.0
    port            2021

service:
    flush: 1
    log_level: info
pipeline:
    inputs:
        - name: fluentbit_metrics
          tag: internal_metrics
          scrape_interval: 2

    outputs:
        - name: prometheus_exporter
          match: internal_metrics
          host: 0.0.0.0
          port: 2021

You can test the expose of the metrics by using curl:

curl http://127.0.0.1:2021/metrics

The exec_wasi input plugin, allows to execute WASM program that is WASI target like as external program and collects event logs from there.

Configuration Parameters

The plugin supports the following configuration parameters:

Configuration Examples

Here is a configuration example. in_exec_wasi can handle parser. To retrieve from structured data from WASM program, you have to create parser.conf:

Note that Time_Format should be aligned for the format of your using timestamp. In this documents, we assume that WASM program should write JSON style strings into stdout.

[PARSER]
    Name        wasi
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L %z

Then, you can specify the above parsers.conf in the main fluent-bit configuration:

[SERVICE]
    Flush        1
    Daemon       Off
    Parsers_File parsers.conf
    Log_Level    info
    HTTP_Server  Off
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name exec_wasi
    Tag  exec.wasi.local
    WASI_Path /path/to/wasi/program.wasm
    Accessible_Paths .,/path/to/accessible
    Parser wasi

[OUTPUT]
    Name  stdout
    Match *

Forward is the protocol used by Fluent Bit and Fluentd to route messages between peers. This plugin implements the input service to listen for Forward messages.

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

In order to receive Forward messages, you can run the plugin from the command line or through the configuration file as shown in the following examples.

Command Line

From the command line you can let Fluent Bit listen for Forward messages with the following options:

$ fluent-bit -i forward -o stdout

By default the service will listen an all interfaces (0.0.0.0) through TCP port 24224, optionally you can change this directly, e.g:

$ fluent-bit -i forward -p listen="192.168.3.2" -p port=9090 -o stdout

In the example the Forward messages will only arrive through network interface under 192.168.3.2 address and TCP Port 9090.

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name              forward
    Listen            0.0.0.0
    Port              24224
    Buffer_Chunk_Size 1M
    Buffer_Max_Size   6M

[OUTPUT]
    Name   stdout
    Match  *

pipeline:
    inputs:
        - name: forward
          listen: 0.0.0.0
          port: 24224
          buffer_chunk_size: 1M
          buffer_max_size: 6M
    outputs:
        - name: stdout
          match: '*'

Fluent Bit + Secure Forward Setup

Since Fluent Bit v3, in_forward can handle secure forward protocol.

For using user-password authentication, it needs to specify security.users at least an one-pair. For using shared key, it needs to specify shared_key in both of forward output and forward input. self_hostname is not able to specify with the same hostname between fluent servers and clients.

[INPUT]
    Name              forward
    Listen            0.0.0.0
    Port              24224
    Buffer_Chunk_Size 1M
    Buffer_Max_Size   6M
    Security.Users fluentbit changeme
    Shared_Key secret
    Self_Hostname flb.server.local

[OUTPUT]
    Name   stdout
    Match  *

pipeline:
    inputs:
        - name: forward
          listen: 0.0.0.0
          port: 24224
          buffer_chunk_size: 1M
          buffer_max_size: 6M
          security.users: fluentbit changeme
          shared_key: secret
          self_hostname: flb.server.local
    outputs:
        - name: stdout
          match: '*'

Testing

Once Fluent Bit is running, you can send some messages using the fluent-cat tool (this tool is provided by Fluentd:

$ echo '{"key 1": 123456789, "key 2": "abcdefg"}' | fluent-cat my_tag

In Fluent Bit we should see the following output:

$ bin/fluent-bit -i forward -o stdout
Fluent-Bit v0.9.0
Copyright (C) Treasure Data

[2016/10/07 21:49:40] [ info] [engine] started
[2016/10/07 21:49:40] [ info] [in_fw] binding 0.0.0.0:24224
[0] my_tag: [1475898594, {"key 1"=>123456789, "key 2"=>"abcdefg"}]

The mem input plugin, gathers the information about the memory and swap usage of the running system every certain interval of time and reports the total amount of memory and the amount of free available.

Getting Started

In order to get memory and swap usage from your system, you can run the plugin from the command line or through the configuration file:

Command Line

Threading

Configuration File

In your main configuration file append the following Input & Output sections:

Configuration Parameters

TLS / SSL

HTTP input plugin supports TLS/SSL, for more details about the properties available and general configuration, please refer to the section.

gzipped content

The HTTP input plugin will accept and automatically handle gzipped content as of v2.2.1 as long as the header Content-Encoding: gzip is set on the received data.

Getting Started

The http input plugin allows Fluent Bit to open up an HTTP port that you can then route data to in a dynamic way. This plugin supports dynamic tags which allow you to send data with different tags through the same input. An example video and curl message can be seen below

How to set tag

The tag for the HTTP input plugin is set by adding the tag to the end of the request URL. This tag is then used to route the event through the system. For example, in the following curl message below the tag set is app.log**. ** because the end end path is /app_log:

Curl request

Configuration File

If you do not set the tag http.0 is automatically used. If you have multiple HTTP inputs then they will follow a pattern of http.N where N is an integer representing the input.

Curl request

Configuration File

How to set tag_key

The tag_key configuration option allows to specify the key name that will be used to overwrite a tag. The tag's value will be replaced with the value associated with the specified key. For example, setting tag_key to "custom_tag" and the log event contains a json field with the key "custom_tag" Fluent Bit will use the value of that field as the new tag for routing the event through the system.

Curl request

Configuration File

How to set multiple custom HTTP header on success

The success_header parameter allows to set multiple HTTP headers on success. The format is:

Example Curl message

Configuration File

Command Line

Health input plugin allows you to check how healthy a TCP server is. It does the check by issuing a TCP connection every a certain interval of time.

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

In order to start performing the checks, you can run the plugin from the command line or through the configuration file:

Command Line

From the command line you can let Fluent Bit generate the checks with the following options:

Configuration File

In your main configuration file append the following Input & Output sections:

Testing

Once Fluent Bit is running, you will see some random values in the output interface similar to this:

The kmsg input plugin reads the Linux Kernel log buffer since the beginning, it gets every record and parse it field as priority, sequence, seconds, useconds, and message.

Configuration Parameters

Getting Started

In order to start getting the Linux Kernel messages, you can run the plugin from the command line or through the configuration file:

Command Line

As described above, the plugin processed all messages that the Linux Kernel reported, the output has been truncated for clarification.

Configuration File

In your main configuration file append the following Input & Output sections:

The Kafka input plugin allows subscribing to one or more Kafka topics to collect messages from an service. This plugin uses the official (built-in dependency).

Configuration Parameters

Getting Started

In order to subscribe/collect messages from Apache Kafka, you can run the plugin from the command line or through the configuration file:

Command Line

The kafka plugin can read parameters through the -p argument (property), e.g:

Configuration File

In your main configuration file append the following Input & Output sections:

Example of using kafka input/output plugins

The Fluent Bit source repository contains a full example of using Fluent Bit to process Kafka records:

The above will connect to the broker listening on kafka-broker:9092 and subscribe to the fb-source topic, polling for new messages every 100 milliseconds.

Since the payload will be in json format, we ask the plugin to automatically parse the payload with format json.

Every message received is then processed with kafka.lua and sent back to the fb-sink topic of the same broker.

The example can be executed locally with make start in the examples/kafka_filter directory (docker/compose is used).

Kubernetes exports it events through the API server. This input plugin allows to retrieve those events as logs and get them processed through the pipeline.

Configuration

• * As of Fluent-Bit 3.1, this plugin uses a Kubernetes watch stream instead of polling. In versions before 3.1, the interval parameters are used for reconnecting the Kubernetes watch stream.

Threading

Getting Started

Kubernetes Service Account

The Kubernetes service account used by Fluent Bit must have get, list, and watch permissions to namespaces and pods for the namespaces watched in the kube_namespace configuration parameter. If you're using the helm chart to configure Fluent Bit, this role is included.

Simple Configuration File

Event Timestamp

Event timestamps are created from the first existing field, based on the following order of precedence:

1. lastTimestamp

2. firstTimestamp

3. metadata.creationTimestamp

The netif input plugin gathers network traffic information of the running system every certain interval of time, and reports them.

The Network I/O Metrics plugin creates metrics that are log-based, such as JSON payload. For Prometheus-based metrics, see the Node Exporter Metrics input plugin.

Configuration Parameters

The plugin supports the following configuration parameters:

Getting Started

In order to monitor network traffic from your system, you can run the plugin from the command line or through the configuration file:

Command Line

$ bin/fluent-bit -i netif -p interface=eth0 -o stdout
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2017/07/08 23:34:18] [ info] [engine] started
[0] netif.0: [1499524459.001698260, {"eth0.rx.bytes"=>89769869, "eth0.rx.packets"=>73357, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>4256474, "eth0.tx.packets"=>24293, "eth0.tx.errors"=>0}]
[1] netif.0: [1499524460.002541885, {"eth0.rx.bytes"=>98, "eth0.rx.packets"=>1, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>98, "eth0.tx.packets"=>1, "eth0.tx.errors"=>0}]
[2] netif.0: [1499524461.001142161, {"eth0.rx.bytes"=>98, "eth0.rx.packets"=>1, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>98, "eth0.tx.packets"=>1, "eth0.tx.errors"=>0}]
[3] netif.0: [1499524462.002612971, {"eth0.rx.bytes"=>98, "eth0.rx.packets"=>1, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>98, "eth0.tx.packets"=>1, "eth0.tx.errors"=>0}]

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name          netif
    Tag           netif
    Interval_Sec  1
    Interval_NSec 0
    Interface     eth0
[OUTPUT]
    Name   stdout
    Match  *

pipeline:
    inputs:
        - name: netif
          tag: netif
          interval_sec: 1
          interval_nsec: 0
          interface: eth0
    outputs:
        - name: stdout
          match: '*'

Note: Total interval (sec) = Interval_Sec + (Interval_Nsec / 1000000000).

e.g. 1.5s = 1s + 500000000ns

Configuration Parameters

Getting Started

The podman metrics input plugin allows Fluent Bit to gather podman container metrics. The entire procedure of collecting container list and gathering data associated with them bases on filesystem data.This plugin does not execute podman commands or send http requests to podman api - instead it reads podman configuration file and metrics exposed by /sys and /proc filesystems.

This plugin supports and automatically detects both cgroups v1 and v2.

Example Curl message for one running container

$> curl 0.0.0.0:2021/metrics
# HELP fluentbit_input_bytes_total Number of input bytes.
# TYPE fluentbit_input_bytes_total counter
fluentbit_input_bytes_total{name="podman_metrics.0"} 0
# HELP fluentbit_input_records_total Number of input records.
# TYPE fluentbit_input_records_total counter
fluentbit_input_records_total{name="podman_metrics.0"} 0
# HELP container_memory_usage_bytes Container memory usage in bytes
# TYPE container_memory_usage_bytes counter
container_memory_usage_bytes{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest"} 884736
# HELP container_cpu_user_seconds_total Container cpu usage in seconds in user mode
# TYPE container_cpu_user_seconds_total counter
container_cpu_user_seconds_total{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest"} 0
# HELP container_cpu_usage_seconds_total Container cpu usage in seconds
# TYPE container_cpu_usage_seconds_total counter
container_cpu_usage_seconds_total{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest"} 0
# HELP container_network_receive_bytes_total Network received bytes
# TYPE container_network_receive_bytes_total counter
container_network_receive_bytes_total{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest",interface="eth0"} 8515
# HELP container_network_receive_errors_total Network received errors
# TYPE container_network_receive_errors_total counter
container_network_receive_errors_total{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest",interface="eth0"} 0
# HELP container_network_transmit_bytes_total Network transmited bytes
# TYPE container_network_transmit_bytes_total counter
container_network_transmit_bytes_total{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest",interface="eth0"} 962
# HELP container_network_transmit_errors_total Network transmitedd errors
# TYPE container_network_transmit_errors_total counter
container_network_transmit_errors_total{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest",interface="eth0"} 0
# HELP fluentbit_input_storage_overlimit Is the input memory usage overlimit ?.
# TYPE fluentbit_input_storage_overlimit gauge
fluentbit_input_storage_overlimit{name="podman_metrics.0"} 0
# HELP fluentbit_input_storage_memory_bytes Memory bytes used by the chunks.
# TYPE fluentbit_input_storage_memory_bytes gauge
fluentbit_input_storage_memory_bytes{name="podman_metrics.0"} 0
# HELP fluentbit_input_storage_chunks Total number of chunks.
# TYPE fluentbit_input_storage_chunks gauge
fluentbit_input_storage_chunks{name="podman_metrics.0"} 0
# HELP fluentbit_input_storage_chunks_up Total number of chunks up in memory.
# TYPE fluentbit_input_storage_chunks_up gauge
fluentbit_input_storage_chunks_up{name="podman_metrics.0"} 0
# HELP fluentbit_input_storage_chunks_down Total number of chunks down.
# TYPE fluentbit_input_storage_chunks_down gauge
fluentbit_input_storage_chunks_down{name="podman_metrics.0"} 0
# HELP fluentbit_input_storage_chunks_busy Total number of chunks in a busy state.
# TYPE fluentbit_input_storage_chunks_busy gauge
fluentbit_input_storage_chunks_busy{name="podman_metrics.0"} 0
# HELP fluentbit_input_storage_chunks_busy_bytes Total number of bytes used by chunks in a busy state.
# TYPE fluentbit_input_storage_chunks_busy_bytes gauge
fluentbit_input_storage_chunks_busy_bytes{name="podman_metrics.0"} 0

Configuration File

[INPUT]
    name podman_metrics
    scrape_interval 10
    scrape_on_start true
[OUTPUT]
    name prometheus_exporter

Command Line

$> fluent-bit -i podman_metrics -o prometheus_exporter

Exposed metrics

Currently supported counters are:

• container_memory_usage_bytes

• container_memory_max_usage_bytes

• container_memory_rss

• container_spec_memory_limit_bytes

• container_cpu_user_seconds_total

• container_cpu_usage_seconds_total

• container_network_receive_bytes_total

• container_network_receive_errors_total

• container_network_transmit_bytes_total

• container_network_transmit_errors_total

This plugin mimics naming convetion of docker metrics exposed by cadvisor project

Prometheus Node Exporter is a popular way to collect system level metrics from operating systems, such as CPU / Disk / Network / Process statistics. Fluent Bit 1.8.0 includes node exporter metrics plugin that builds off the Prometheus design to collect system level metrics without having to manage two separate processes or agents.

The initial release of Node Exporter Metrics contains a subset of collectors and metrics available from Prometheus Node Exporter and we plan to expand them over time.

Important note: Metrics collected with Node Exporter Metrics flow through a separate pipeline from logs and current filters do not operate on top of metrics.

This plugin is supported on Linux-based operating systems for the most part with macOS offering a reduced subset of metrics. The table below indicates which collector is supported on macOS.

Configuration

Note: The plugin top-level scrape_interval setting is the global default with any custom settings for individual scrape_intervals then overriding just that specific metric scraping interval. Each collector.xxx.scrape_interval option only overrides the interval for that specific collector and updates the associated set of provided metrics.

The overridden intervals only change the collection interval, not the interval for publishing the metrics which is taken from the global setting. For example, if the global interval is set to 5s and an override interval of 60s is used then the published metrics will be reported every 5s but for the specific collector they will stay the same for 60s until it is collected again. This feature aims to help with down-sampling when collecting metrics.

Collectors available

The following table describes the available collectors as part of this plugin. All of them are enabled by default and respects the original metrics name, descriptions, and types from Prometheus Exporter, so you can use your current dashboards without any compatibility problem.

note: the Version column specifies the Fluent Bit version where the collector is available.

Threading

This input always runs in its own thread.

Getting Started

Simple Configuration File

In the following configuration file, the input plugin _node_exporter_metrics collects _metrics every 2 seconds and exposes them through our Prometheus Exporter output plugin on HTTP/TCP port 2021.

# Node Exporter Metrics + Prometheus Exporter
# -------------------------------------------
# The following example collect host metrics on Linux and expose
# them through a Prometheus HTTP end-point.
#
# After starting the service try it with:
#
# $ curl http://127.0.0.1:2021/metrics
#
[SERVICE]
    flush           1
    log_level       info

[INPUT]
    name            node_exporter_metrics
    tag             node_metrics
    scrape_interval 2

[OUTPUT]
    name            prometheus_exporter
    match           node_metrics
    host            0.0.0.0
    port            2021

# Node Exporter Metrics + Prometheus Exporter
# -------------------------------------------
# The following example collect host metrics on Linux and expose
# them through a Prometheus HTTP end-point.
#
# After starting the service try it with:
#
# $ curl http://127.0.0.1:2021/metrics
#
service:
    flush: 1
    log_level: info
pipeline:
    inputs:
        - name: node_exporter_metrics
          tag:  node_metrics
          scrape_interval: 2
    outputs:
        - name: prometheus_exporter
          match: node_metrics
          host: 0.0.0.0
          port: 2021