Elasticsearch
Send logs to Elasticsearch (including Amazon OpenSearch Service)
The Elasticsearch (es) output plugin lets you ingest your records into an Elasticsearch database. To use this plugin, you must have an operational Elasticsearch service running in your environment.
In Fluent Bit 5.1 and later, elasticsearch is accepted as an alias for the plugin name es. Both name: es and name: elasticsearch load this output plugin. The alias applies only to output plugins. The Elasticsearch input plugin is named elasticsearch and is unaffected.
Configuration parameters
aws_auth
Enable AWS Sigv4 Authentication for Amazon OpenSearch Service.
Off
aws_external_id
External ID for the AWS IAM Role specified with aws_role_arn.
none
aws_profile
AWS profile name.
none
aws_region
Specify the AWS region for Amazon OpenSearch Service.
none
aws_role_arn
AWS IAM Role to assume to put records to your Amazon cluster.
none
aws_service_name
Service name to use in AWS Sigv4 signature. For integration with Amazon OpenSearch Serverless, set to aoss. See Amazon OpenSearch Serverless for more information.
es
aws_sts_endpoint
Specify the custom STS endpoint to be used with STS API for Amazon OpenSearch Service.
none
buffer_size
Specify the buffer size used to read the response from the Elasticsearch HTTP service. Use for debugging purposes where required to read full responses. Response size grows depending of the number of records inserted. To use an unlimited amount of memory, set this value to False. Otherwise set the value according to the Unit Size.
512k
cloud_auth
Specify the credentials to use to connect to Elastic's Elasticsearch Service running on Elastic Cloud.
none
cloud_id
If using Elastic's Elasticsearch Service you can specify the cloud_id of the cluster running. The string has the format <deployment_name>:<base64_info>. Once decoded, the base64_info string has the format <deployment_region>$<elasticsearch_hostname>$<kibana_hostname>.
none
compress
Set payload compression mechanism. Option available is gzip.
none
current_time_index
Use current time for index generation instead of message record.
Off
drop_unrecoverable_records
Drop records that the _bulk API rejects with an unrecoverable error instead of retrying them. Enabling this option causes data loss. See Bulk retries. Supported in v5.1.3 or later.
Off
generate_id
When enabled, generate _id for outgoing records. This prevents duplicate records when retrying ES. See Bulk retries.
Off
host
IP address or hostname of the target Elasticsearch instance.
127.0.0.1
http_api_key
API key for authenticating with Elasticsearch. Must be base64 encoded. If http_user or cloud_auth are defined, this parameter is ignored.
none
http_passwd
Password for user defined in http_user.
none
http_user
Optional username credential for Elastic X-Pack access.
none
id_key
If set, _id will be the value of the key from the incoming record and generate_id option is ignored.
none
include_tag_key
When enabled, appends the Tag name to the record.
Off
index
Index name.
fluent-bit
logstash_dateformat
Time format based on strftime to generate the second part of the Index name.
%Y.%m.%d
logstash_format
Enable Logstash format compatibility. This option takes a Boolean value: True/False, On/Off.
Off
logstash_prefix
When logstash_format is enabled, the Index name is composed using a prefix and the date. For example, if logstash_prefix is equal to mydata your index will become mydata-YYYY.MM.DD. The last string appended belongs to the date when the data is being generated.
logstash
logstash_prefix_key
When included, the value of the key in the record is evaluated as a key reference and overrides logstash_prefix for index generation. If the key/value isn't found in the record then the logstash_prefix option acts as a fallback. The parameter is expected to be a record accessor.
none
logstash_prefix_separator
Set a separator between logstash_prefix and date.
-
path
Elasticsearch accepts new data on HTTP query path /_bulk. You can also serve Elasticsearch behind a reverse proxy on a sub-path. Define the path by adding a path prefix in the indexing HTTP POST URI.
none
pipeline
Define which pipeline the database should use. For performance reasons, it's strongly suggested to do parsing and filtering on Fluent Bit side, and avoid pipelines.
none
port
TCP port of the target Elasticsearch instance.
9200
replace_dots
When enabled, replace field name dots with underscore. Required by Elasticsearch 2.0-2.3.
Off
suppress_type_name
When enabled, mapping types is removed and type option is ignored. Elasticsearch 8.0.0 or higher no longer supports mapping types.
Off
tag_key
When include_tag_key is enabled, this property defines the key name for the tag.
flb-key
time_key
When logstash_format is enabled, each record will get a new timestamp field. The time_key property defines the name of that field.
@timestamp
time_key_format
When logstash_format is enabled, this property defines the format of the timestamp.
%Y-%m-%dT%H:%M:%S
time_key_nanos
When logstash_format is enabled, enabling this property sends nanosecond precision timestamps.
Off
trace_error
Log diagnostics whenever a bulk request doesn't fully succeed, either because Elasticsearch reported failed operations or because Fluent Bit couldn't interpret the bulk response. Logs the request payload at debug level and the Elasticsearch response at error level.
Off
trace_output
Print all Elasticsearch API request payloads to stdout for diagnostics.
Off
type
Type name.
_doc
write_operation
Operation type for records. Can be any of: create, index, update, upsert.
create
If you have used a common relational database, the parameters index and type can be compared to the database and table concepts.
TLS / SSL
The Elasticsearch output plugin supports TLS/SSL. For more details about the properties available and general configuration, see TLS/SSL.
write_operation
The write_operation can be any of:
create
Adds new data. If the data already exists (based on its id), the op is skipped.
index
New data is added while existing data (based on its id) is replaced (reindexed).
update
Updates existing data (based on its id). If no data is found, the op is skipped.
upsert
Merge or insert if the data doesn't exist, updates if the data exists (based on its id).
Bulk retries
Fluent Bit sends records to the Elasticsearch _bulk API in batches. Elasticsearch can accept some documents in a batch while rejecting others, and reports this with an HTTP 200 response that contains "errors": true and a per-document status in the items array.
When this happens, Fluent Bit inspects each item and rebuilds the payload so that it contains only the documents that weren't accepted. If any documents remain after this filtering, the flush is retried and the next attempt sends that reduced payload instead of the original batch. If no documents remain, Fluent Bit treats the flush as successful and doesn't retry it. Documents that Elasticsearch already accepted aren't sent again, which limits duplicates caused by partial failures. This filtering is always active and has no configuration option, but drop_unrecoverable_records controls what happens to the documents that remain.
A document counts as accepted when its status is in the 2xx range. A 409 conflict also counts as accepted, but only for the create operation, where the conflict confirms the document already exists. A 409 returned for any other operation is treated as a failure and retried.
By default, every document that wasn't accepted stays in the retry payload, including documents Elasticsearch rejected for a reason that won't change on a later attempt, such as a mapping conflict. These documents are retried like any other failure, subject to the Retry_Limit set for the output. In Fluent Bit version 5.1.3 and greater, you can set drop_unrecoverable_records to On to drop them instead of retrying them. Fluent Bit then treats a failed document as unrecoverable when its status is in the 4xx range, except for 408, 409, and 429. The 408 and 429 statuses, and every 5xx status, are still retried. A 409 conflict is never dropped: with the create operation it counts as accepted, and with any other operation it's retried. When every failed document in a batch is dropped this way, nothing remains to send, so Fluent Bit treats the flush as successful instead of retrying it.
Fluent Bit version 5.1.3 and greater logs an error summary whenever a bulk response reports failures. The summary reports how many documents failed, the status, type, and reason of the first error, and how many documents Fluent Bit is retrying. When drop_unrecoverable_records is enabled and documents were dropped, the summary also reports the dropped count, and those documents are added to the output's dropped records metric.
Enabling drop_unrecoverable_records discards the rejected documents permanently. Use it when repeated retries of documents Elasticsearch won't accept are causing a backlog, and you accept losing those documents.
If Fluent Bit can't interpret the bulk response, it logs invalid Elasticsearch bulk response and retries the entire batch. Set trace_error to On to log the request and response for diagnosis. Because a batch can still be re-sent in full in this case, set generate_id or id_key so that every document carries a stable, unique _id across attempts. What a repeated write then does depends on write_operation. With create, Elasticsearch rejects the duplicate with a 409 conflict, which Fluent Bit counts as accepted. With index, update, or upsert, Elasticsearch applies the write again, and these operations aren't inherently no-ops, so choose an _id scheme and a write operation that make a repeated write safe for your data.
Get started
To insert records into an Elasticsearch service, run the plugin from the command line or through the configuration file.
Command line
The es plugin can read the parameters from the command line in the following ways:
Through the
-pargument (property).Setting them directly through the service URI.
The URI format is the following:
Using the format specified, you could start Fluent Bit through:
Which is similar to the following command:
Configuration file
In your main configuration file append the following sections.
Elasticsearch field names
Some input plugins can generate messages where the field names contains dots (.). For Elasticsearch 2.0, this isn't allowed. The current es plugin replaces a dot with an underscore (_):
becomes
Use Fluent Bit ElasticSearch plugin with other services
Connect to Amazon OpenSearch or Elastic Cloud with the ElasticSearch plugin.
Amazon OpenSearch Service
The Amazon OpenSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon OpenSearch Service with IAM Authentication.
See details on how AWS credentials are fetched.
Example configuration:
Be aware that the Port is set to 443, tls is enabled, and AWS_Region is set.
Use Fluent Bit with Elastic Cloud
Fluent Bit supports connecting to Elastic Cloud by providing the cloud_id and the cloud_auth settings. cloud_auth uses the elastic user and password provided when the cluster was created. For details refer to the Cloud ID usage page.
Example configuration:
In Elastic Cloud version 8 and great, the type option must be removed by setting Suppress_Type_Name On.
Without this you will see errors like:
Troubleshooting
Use the following information to help resolve errors using the ElasticSearch plugin.
Using multiple types in a single index
Elasticsearch 6.0 can't create multiple types in a single index. An error message like the following indicates you need to update your configuration to use a single type on each index.
This means that you can't set up your configuration like the following:.
For details, read the official blog post on that issue.
Mapping type names can't start with underscores (_)
Fluent Bit v1.5 changed the default mapping type from flb_type to _doc, matching the recommendation from Elasticsearch for version 6.2 and greater (see commit with rationale).
This doesn't work in Elasticsearch versions 5.6 through 6.1 (discussion and fix).
Ensure you set an explicit map such as doc or flb_type in the configuration, as seen on the last line:
Validation failures
In Fluent Bit v1.8.2 and greater, Fluent Bit started using create method (instead of index) for data submission. This makes Fluent Bit compatible with Datastream, introduced in Elasticsearch 7.9. You might see errors like:
If you see action_request_validation_exception errors on your pipeline with Fluent Bit versions greater than v1.8.2, correct them by turning on Generate_ID as follows:
Logstash_Prefix_Key
The following snippet demonstrates using the namespace name as extracted by the kubernetes filter as logstash prefix:
For records that don't have the field kubernetes.namespace_name, the default prefix logstash will be used.
Last updated
Was this helpful?