Grep - Fluent Bit: Official Manual

The Grep Filter plugin allows you to match or exclude specific records based on regular expression patterns for values or nested values.
Configuration Parameters
The plugin supports the following configuration parameters:
Key
Value Format
Description
Regex
KEY REGEX
Keep records in which the content of KEY matches the regular expression.
Exclude
KEY REGEX
Exclude records in which the content of KEY matches the regular expression.
Record Accessor Enabled
This plugin enables the Record Accessor feature to specify the KEY. Using the record accessor is suggested if you want to match values against nested values.
Getting Started
In order to start filtering records, you can run the filter from the command line or through the configuration file. The following example assumes that you have a file called lines.txt with the following content:
1
{"log": "aaa"}
2
{"log": "aab"}
3
{"log": "bbb"}
4
{"log": "ccc"}
5
{"log": "ddd"}
6
{"log": "eee"}
7
{"log": "fff"}
8
{"log": "ggg"}
Copied!
Command Line
Note: using the command line mode need special attention to quote the regular expressions properly. It's suggested to use a configuration file.
The following command will load the tail plugin and read the content of lines.txt file. Then the grep filter will apply a regular expression rule over the log field (created by tail plugin) and only pass the records which field value starts with aa:
1
$ bin/fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o stdout
Copied!
Configuration File
1
[INPUT]
2
    name   tail
3
    path   lines.txt
4
    parser json
5
​
6
[FILTER]
7
    name   grep
8
    match  *
9
    regex  log aa
10
​
11
[OUTPUT]
12
    name   stdout
13
    match  *
Copied!
The filter allows to use multiple rules which are applied in order, you can have many Regex and Exclude entries as required.
Nested fields example
If you want to match or exclude records based on nested values, you can use a Record Accessor format as the KEY name. Consider the following record example:
1
{
2
    "log": "something",
3
    "kubernetes": {
4
        "pod_name": "myapp-0",
5
        "namespace_name": "default",
6
        "pod_id": "216cd7ae-1c7e-11e8-bb40-000c298df552",
7
        "labels": {
8
            "app": "myapp"
9
        },
10
        "host": "minikube",
11
        "container_name": "myapp",
12
        "docker_id": "370face382c7603fdd309d8c6aaaf434fd98b92421ce"
13
    }
14
}
Copied!
if you want to exclude records that match given nested field (for example kubernetes.labels.app), you can use the following rule:
1
[FILTER]
2
    Name    grep
3
    Match   *
4
    Exclude $kubernetes['labels']['app'] myapp
Copied!
Excluding records missing/invalid fields
It may be that in your processing pipeline you want to drop records that are missing certain keys.
A simple way to do this is just to exclude with a regex that matches anything, a missing key will fail this check.
Here is an example that checks for a specific valid value for the key as well:
1
# Use Grep to verify the contents of the iot_timestamp value.
2
# If the iot_timestamp key does not exist, this will fail
3
# and exclude the row.
4
[FILTER]
5
    Name                     grep
6
    Alias                    filter-iots-grep
7
    Match                    iots_thread.*
8
    Regex                    iot_timestamp ^\d{4}-\d{2}-\d{2}
Copied!
The specified key iot_timestamp must match the expected expression - if it does not or is missing/empty then it will be excluded.

GeoIP2 Filter

Kubernetes

Last modified 2mo ago

Export as PDF

Copy link

Contents

Configuration Parameters

Getting Started

Command Line

Configuration File

Nested fields example

Excluding records missing/invalid fields