Elasticsearch
Send logs to Elasticsearch (including Amazon Elasticsearch Service)
The es output plugin, allows to ingest your records into a Elasticsearch database. The following instructions assumes that you have a fully operational Elasticsearch service running in your environment.
Configuration Parameters
The parameters index and type can be confusing if you are new to Elastic, if you have used a common relational database before, they can be compared to the database and table concepts. Also see the FAQ below
TLS / SSL
Elasticsearch output plugin supports TTL/SSL, for more details about the properties available and general configuration, please refer to the TLS/SSL section.
Getting Started
In order to insert records into a Elasticsearch service, you can run the plugin from the command line or through the configuration file:
Command Line
The es plugin, can read the parameters from the command line in two ways, through the -p argument (property) or setting them directly through the service URI. The URI format is the following:
Using the format specified, you could start Fluent Bit through:
which is similar to do:
Configuration File
In your main configuration file append the following Input & Output sections. You can visualize this configuration here
About Elasticsearch field names
Some input plugins may generate messages where the field names contains dots, since Elasticsearch 2.0 this is not longer allowed, so the current es plugin replaces them with an underscore, e.g:
becomes
FAQ
Elasticsearch rejects requests saying "the final mapping would have more than 1 type"
Since Elasticsearch 6.0, you cannot create multiple types in a single index. This means that you cannot set up your configuration as below anymore.
If you see an error message like below, you'll need to fix your configuration to use a single type on each index.
Rejecting mapping update to [search] as the final mapping would have more than 1 type
For details, please read the official blog post on that issue.
Elasticsearch rejects requests saying "Document mapping type name can't start with '_'"
Fluent Bit v1.5 changed the default mapping type from flb_type
to _doc
, which matches the recommendation from Elasticsearch from version 6.2 forwards (see commit with rationale). This doesn't work in Elasticsearch versions 5.6 through 6.1 (see Elasticsearch discussion and fix). Ensure you set an explicit map (such as doc
or flb_type
) in the configuration, as seen on the last line:
Fluent Bit + Amazon Elasticsearch Service
The Amazon ElasticSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon ElasticSearch Service with IAM Authentication.
Fluent Bit supports sourcing AWS credentials from any of the standard sources (for example, an Amazon EKS IAM Role for a Service Account).
Example configuration:
Notice that the Port
is set to 443
, tls
is enabled, and AWS_Region
is set.
Last updated