Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Look up Geo data from IP
GeoIP2 Filter allows you to enrich the incoming data stream using location data from GeoIP2 database.
This plugin supports the following configuration parameters:
database
Path to the GeoIP2 database.
lookup_key
Field name to process
record
Defines the KEY LOOKUP_KEY VALUE
triplet. See below for how to set up this option.
The following configuration will process incoming remote_addr
, and append country information retrieved from GeoLite2 database.
Each Record
parameter above specifies the following triplet:
The field name to be added to records (country
)
The lookup key to process (remote_addr
)
The query for GeoIP2 database (%{country.names.en}
)
By running Fluent Bit with the configuration above, you will see the following output:
Note that the GeoLite2-City.mmdb
database is available from MaxMind's official site.
The following plugin looks up if a value in a specified list exists and then allows the addition of a record to indicate if found. Introduced in version 1.8.4
The plugin supports the following configuration parameters
file
The single value file that Fluent Bit will use as a lookup table to determine if the specified lookup_key
exists
lookup_key
The specific key to look up and determine if it exists, supports record accessor
record
The record to add if the lookup_key
is found in the specified file
. Note you may add multiple record parameters.
mode
Set the check mode. exact
and partial
are supported. Default : exact
.
print_query_time
Print to stdout the elapseed query time for every matched record. Default: false
ignore_case
Compare strings by ignoring case. Default: false
In the following configuration we will read a file test1.log
that includes the following values
Additionally, we will use the following lookup file which contains a list of malicious IPs (ip_list.txt
)
In the configuration we are using $remote_addr as the lookup key and 7.7.7.7 is malicious. This means the record we would output for the last record would look like the following
The AWS Filter Enriches logs with AWS Metadata. Currently the plugin adds the EC2 instance ID and availability zone to log records. To use this plugin, you must be running in EC2 and have the instance metadata service enabled.
The plugin supports the following configuration parameters:
imds_version
Specify which version of the instance metadata service to use. Valid values are 'v1' or 'v2'.
v2
az
true
ec2_instance_id
The EC2 instance ID.
true
ec2_instance_type
The EC2 instance type.
false
private_ip
The EC2 instance private ip.
false
ami_id
The EC2 instance image id.
false
account_id
The account ID for current EC2 instance.
false
hostname
The hostname for current EC2 instance.
false
vpc_id
The VPC ID for current EC2 instance.
false
tags_enabled
false
tags_include
Defines list of specific EC2 tag keys to inject into the logs. Tag keys must be separated by "," character. Tags which are not present in this list will be ignored. Example: Name,tag1,tag2
.
tags_exclude
Defines list of specific EC2 tag keys not to inject into the logs. Tag keys must be separated by "," character. Tags which are not present in this list will be injected into the logs. If both tags_include
and tags_exclude
are specified, configuration is invalid and plugin fails. Example: Name,tag1,tag2
retry_interval_s
Defines minimum duration between retries for fetching EC2 instance tags.
300
Note: If you run Fluent Bit in a container, you may have to use instance metadata v1. The plugin behaves the same regardless of which version is used.
EC2 Tags are a useful feature that enables you to label and organize your EC2 instances by creating custom-defined key-value pairs. These tags are commonly utilized for resource management, cost allocation, and automation. Consequently, including them in the Fluent Bit generated logs is almost essential.
To achieve this, AWS Filter can be configured with tags_enabled true
to enable the tagging of logs with the relevant EC2 instance tags. This setup ensures that logs are appropriately tagged, making it easier to manage and analyze them based on specific criteria.
To use the tags_enabled true
functionality in Fluent Bit, the instance-metadata-tags option must be enabled on the EC2 instance where Fluent Bit is running. Without this option enabled, Fluent Bit will not be able to retrieve the tags associated with the EC2 instance. However, this does not mean that Fluent Bit will fail or stop working altogether. Instead, if instance-metadata-tags option is not enabled, Fluent Bit will continue to operate normally and capture other values, such as the EC2 instance ID or availability zone, based on its configuration.
Assume that our EC2 instance has many tags, some of which have lengthy values that are irrelevant to the logs we want to collect. Only two tags, department
and project
, seem to be valuable for our purpose. Here is a configuration which reflects this requirement:
If we run Fluent Bit, what will the logs look like? Here is an example of what the logs might contain:
Suppose our EC2 instance has three tags: Name:fluent-bit-docs-example
, project:fluentbit
, and department:it
. In this example, we want to exclude the department
tag since we consider it redundant. This is because all of our projects belong to the it
department, and we do not need to waste storage space on redundant labels.
Here is an example configuration that achieves this:
The resulting logs might look like this:
The ; for example, "us-east-1a".
Specifies if should attach EC2 instance tags. EC2 instance must have the option enabled (which is disabled by default).
Made for testing: make sure that your records contain the expected key and values
The expect filter plugin allows you to validate that records match certain criteria in their structure, like validating that a key exists or it has a specific value.
The following page just describes the configuration properties available, for a detailed explanation of its usage and use cases, please refer the following page:
The plugin supports the following configuration parameters:
key_exists
Check if a key with a given name exists in the record.
key_not_exists
Check if a key does not exist in the record.
key_val_is_null
check that the value of the key is NULL.
key_val_is_not_null
check that the value of the key is NOT NULL.
key_val_eq
check that the value of the key equals the given value in the configuration.
action
action to take when a rule does not match. The available options are warn
, exit
or "result_key". On warn
, a warning message is sent to the logging layer when a mismatch of the rules above is found; using exit
makes Fluent Bit abort with status code 255
; result_key
is to add a matching result to each record.
result_key
specify a key name of matching result. This key is to be used only when 'action' is 'result_key'.
As mentioned on top, refer to the following page for specific details of usage of this filter: