Elasticsearch
Send logs to Elasticsearch (including Amazon OpenSearch Service)
The es output plugin, allows to ingest your records into an Elasticsearch database. The following instructions assumes that you have a fully operational Elasticsearch service running in your environment.
Configuration Parameters
The parameters index and type can be confusing if you are new to Elastic, if you have used a common relational database before, they can be compared to the database and table concepts. Also see the FAQ below
TLS / SSL
Elasticsearch output plugin supports TTL/SSL, for more details about the properties available and general configuration, please refer to the TLS/SSL section.
Getting Started
In order to insert records into a Elasticsearch service, you can run the plugin from the command line or through the configuration file:
Command Line
The es plugin, can read the parameters from the command line in two ways, through the -p argument (property) or setting them directly through the service URI. The URI format is the following:
Using the format specified, you could start Fluent Bit through:
which is similar to do:
Configuration File
In your main configuration file append the following Input & Output sections. You can visualize this configuration here
About Elasticsearch field names
Some input plugins may generate messages where the field names contains dots, since Elasticsearch 2.0 this is not longer allowed, so the current es plugin replaces them with an underscore, e.g:
becomes
FAQ
Elasticsearch rejects requests saying "the final mapping would have more than 1 type"
Since Elasticsearch 6.0, you cannot create multiple types in a single index. This means that you cannot set up your configuration as below anymore.
If you see an error message like below, you'll need to fix your configuration to use a single type on each index.
Rejecting mapping update to [search] as the final mapping would have more than 1 type
For details, please read the official blog post on that issue.
Elasticsearch rejects requests saying "Document mapping type name can't start with '_'"
Fluent Bit v1.5 changed the default mapping type from flb_type
to _doc
, which matches the recommendation from Elasticsearch from version 6.2 forwards (see commit with rationale). This doesn't work in Elasticsearch versions 5.6 through 6.1 (see Elasticsearch discussion and fix). Ensure you set an explicit map (such as doc
or flb_type
) in the configuration, as seen on the last line:
Fluent Bit + Amazon OpenSearch Service
The Amazon OpenSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon OpenSearch Service with IAM Authentication.
See here for details on how AWS credentials are fetched.
Example configuration:
Notice that the Port
is set to 443
, tls
is enabled, and AWS_Region
is set.
Fluent Bit + Elastic Cloud
Fluent Bit supports connecting to Elastic Cloud providing just the cloud_id
and the cloud_auth
settings.
Example configuration:
Validation Failed: 1: an id must be provided if version type or value are set
Since v1.8.2, Fluent Bit started using create
method (instead of index
) for data submission. This makes Flunt Bit compatible with Datastream introduced in Elasticsearch 7.9.
If you see action_request_validation_exception
errors on your pipeline with Fluent Bit >= v1.8.2, you can fix it up by turning on Generate_ID
as follows:
Last updated