Elasticsearch

Supported event types: logs

The Elasticsearch input plugin handles both Elasticsearch and OpenSearch Bulk API requests.

Configuration parameters

The plugin supports the following configuration parameters:

The table below includes both:

• settings specific to the Elasticsearch input plugin

• shared http_server.* listener settings that are used by several HTTP-based inputs

For a cross-plugin explanation of the shared listener settings, see Shared HTTP listener settings for inputs.

Key	Description	Default value
buffer_chunk_size	Set the buffer chunk size. Compatibility alias for http_server.buffer_chunk_size.	512K
buffer_max_size	Set the maximum size of buffer. Compatibility alias for http_server.buffer_max_size.	4M
hostname	Specify hostname or fully qualified domain name. This parameter can be used for "sniffing" (auto-discovery of) cluster node information.	localhost
http2	Enable HTTP/2 support. Compatibility alias for http_server.http2.	true
http_server.max_connections	Maximum number of concurrent active HTTP connections. 0 means unlimited.	0
http_server.workers	Number of HTTP listener worker threads.	1
http_server.ingress_queue_event_limit	Maximum number of deferred ingress queue entries. Applies only when http_server.workers is greater than 1.	8192
http_server.ingress_queue_byte_limit	Maximum size of the deferred ingress queue. Applies only when http_server.workers is greater than 1.	256M
listen	The address to listen on.	0.0.0.0
meta_key	Specify a key name for meta information.	@meta
port	The port for Fluent Bit to listen on.	9200
tag_key	Specify a key name for extracting as a tag.	NULL
threaded	Indicates whether to run this input in its own thread.	false
version	Specify the Elasticsearch version that Fluent Bit reports to clients during sniffing and API requests.	8.0.0

The http_server.ingress_queue_event_limit and http_server.ingress_queue_byte_limit settings matter only when http_server.workers is greater than 1.

TLS / SSL

The Elasticsearch input plugin supports TLS/SSL for receiving data from Beats agents or other clients over encrypted connections. For more details about the properties available and general configuration, refer to Transport Security.

When configuring TLS for Elasticsearch ingestion, common options include:

• tls.verify: Enable or disable certificate validation for incoming connections.

• tls.ca_file: Specify a CA certificate to validate client certificates when using mutual TLS (mTLS).

• tls.crt_file and tls.key_file: Provide the server certificate and private key.

Sniffing

Elasticsearch clients use a process called "sniffing" to automatically discover cluster nodes. When a client connects, it can query the cluster to retrieve a list of available nodes and their addresses. This allows the client to distribute requests across the cluster and adapt when nodes join or leave.

The hostname parameter specifies the hostname or fully qualified domain name that Fluent Bit returns during sniffing requests. Clients use this information to build their connection list. Set this value to match how clients should reach this Fluent Bit instance (for example, an external IP or load balancer address rather than localhost in production environments).

Get started

In order to start performing the checks, you can run the plugin from the command line or through the configuration file:

Command line

From the command line you can configure Fluent Bit to handle Bulk API requests with the following options:

fluent-bit -i elasticsearch -p port=9200 -o stdout

Configuration file

In your configuration file append the following:

fluent-bit.yaml

pipeline:
  inputs:
    - name: elasticsearch
      listen: 0.0.0.0
      port: 9200

  outputs:
    - name: stdout
      match: '*'

fluent-bit.conf

As described previously, the plugin will handle ingested Bulk API requests. For large bulk ingestion, you might have to increase buffer size using the buffer_max_size and buffer_chunk_size parameters:

fluent-bit.yaml

pipeline:
  inputs:
    - name: elasticsearch
      listen: 0.0.0.0
      port: 9200
      buffer_max_size: 20M
      buffer_chunk_size: 5M

  outputs:
    - name: stdout
      match: '*'

fluent-bit.conf

Ingesting from beats series

Ingesting from beats series agents is also supported. For example, Filebeats, Metricbeat, and Winlogbeat are able to ingest their collected data through this plugin.

The Fluent Bit node information is returning as Elasticsearch 8.0.0.

Users must specify the following configurations on their beats configurations:

output.elasticsearch:
  allow_older_versions: true
  ilm: false

For large log ingestion on these beat plugins, users might have to configure rate limiting on those beats plugins when Fluent Bit indicates that the application is exceeding the size limit for HTTP requests:

processors:
  - rate_limit:
      limit: "200/s"