Kafka
Kafka output plugin allows to ingest your records into an Apache Kafka service. This plugin use the official librdkafka C library (built-in dependency)
Configuration Parameters
Setting
rdkafka.log.connection.close
tofalse
andrdkafka.request.required.acks
to 1 are examples of recommended settings of librdfkafka properties.
Getting Started
In order to insert records into Apache Kafka, you can run the plugin from the command line or through the configuration file:
Command Line
The kafka plugin can read parameters through the -p argument (property), e.g:
Configuration File
In your main configuration file append the following Input & Output sections:
Avro Support
Fluent-bit comes with support for avro encoding for the out_kafka plugin. Avro support is optional and must be activated at build-time by using a build def with cmake: -DFLB_AVRO_ENCODER=On
such as in the following example which activates:
out_kafka with avro encoding
fluent-bit's prometheus
metrics via an embedded http endpoint
debugging support
builds the test suites
Kafka Configuration File with Avro Encoding
This is example fluent-bit config tails kubernetes logs, decorates the log lines with kubernetes metadata via the kubernetes filter, and then sends the fully decorated log lines to a kafka broker encoded with a specific avro schema.
Last updated