# Grep The *Grep Filter* plugin lets you match or exclude specific records based on regular expression patterns for values or nested values. ## Configuration parameters The plugin supports the following configuration parameters: | Key | Value Format | Description | | ------------ | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Regex` | KEY REGEX | Keep records where the content of KEY matches the regular expression. | | `Exclude` | KEY REGEX | Exclude records where the content of KEY matches the regular expression. | | `Logical_Op` | Operation | Specify a logical operator: `AND`, `OR` or `legacy` (default). In `legacy` mode the behaviour is either `AND` or `OR` depending on whether the `grep` is including (uses AND) or excluding (uses OR). Available from 2.1 or higher. | ### Record Accessor Enabled Enable the [Record Accessor](/manual/3.2/administration/configuring-fluent-bit/classic-mode/record-accessor.md) feature to specify the KEY. Use the record accessor to match values against nested values. ## Filter records To start filtering records, run the filter from the command line or through the configuration file. The following example assumes that you have a file named `lines.txt` with the following content: ``` {"log": "aaa"} {"log": "aab"} {"log": "bbb"} {"log": "ccc"} {"log": "ddd"} {"log": "eee"} {"log": "fff"} {"log": "ggg"} ``` ### Command line When using the command line, pay close attention to quote the regular expressions. Using a configuration file might be easier. The following command loads the [tail](https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/inputs/tail/README.md) plugin and reads the content of `lines.txt`. Then the `grep` filter applies a regular expression rule over the `log` field created by the `tail` plugin and only passes records with a field value starting with `aa`: ``` $ bin/fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o stdout ``` ### Configuration file {% tabs %} {% tab title="fluent-bit.conf" %} ```python [SERVICE] parsers_file /path/to/parsers.conf [INPUT] name tail path lines.txt parser json [FILTER] name grep match * regex log aa [OUTPUT] name stdout match * ``` {% endtab %} {% tab title="fluent-bit.yaml" %} ```yaml service: parsers_file: /path/to/parsers.conf pipeline: inputs: - name: tail path: lines.txt parser: json filters: - name: grep match: '*' regex: log aa outputs: - name: stdout match: '*' ``` {% endtab %} {% endtabs %} The filter lets you use multiple rules which are applied in order. You can have as many `Regex` and `Exclude` entries as required. ### Nested fields example To match or exclude records based on nested values, you can use [Record Accessor](/manual/3.2/administration/configuring-fluent-bit/classic-mode/record-accessor.md) format as the `KEY` name. Consider the following record example: ```javascript { "log": "something", "kubernetes": { "pod_name": "myapp-0", "namespace_name": "default", "pod_id": "216cd7ae-1c7e-11e8-bb40-000c298df552", "labels": { "app": "myapp" }, "host": "minikube", "container_name": "myapp", "docker_id": "370face382c7603fdd309d8c6aaaf434fd98b92421ce" } } ``` For example, to exclude records that match the nested field `kubernetes.labels.app`, use the following rule: {% tabs %} {% tab title="fluent-bit.conf" %} ```python [FILTER] Name grep Match * Exclude $kubernetes['labels']['app'] myapp ``` {% endtab %} {% tab title="fluent-bit.yaml" %} ```yaml filters: - name: grep match: '*' exclude: $kubernetes['labels']['app'] myapp ``` {% endtab %} {% endtabs %} ### Excluding records with missing or invalid fields You might want to drop records that are missing certain keys. One way to do this is to `exclude` with a regex that matches anything. A missing key fails this check. The followinfg example checks for a specific valid value for the key: {% tabs %} {% tab title="fluent-bit.conf" %} ``` # Use Grep to verify the contents of the iot_timestamp value. # If the iot_timestamp key does not exist, this will fail # and exclude the row. [FILTER] Name grep Alias filter-iots-grep Match iots_thread.* Regex iot_timestamp ^\d{4}-\d{2}-\d{2} ``` {% endtab %} {% tab title="fluent-bit.yaml" %} ```yaml filters: - name: grep alias: filter-iots-grep match: iots_thread.* regex: iot_timestamp ^\d{4}-\d{2}-\d{2} ``` {% endtab %} {% endtabs %} The specified key `iot_timestamp` must match the expected expression. If it doesn't, or is missing or empty, then it will be excluded. ### Multiple conditions If you want to set multiple `Regex` or `Exclude`, use the `Logical_Op` property to use a logical conjuction or disjunction. If `Logical_Op` is set, setting both `Regex` and `Exclude` results in an error. {% tabs %} {% tab title="fluent-bit.conf" %} ```python [INPUT] Name dummy Dummy {"endpoint":"localhost", "value":"something"} Tag dummy [FILTER] Name grep Match * Logical_Op or Regex value something Regex value error [OUTPUT] Name stdout ``` {% endtab %} {% tab title="fluent-bit.yaml" %} ```yaml pipeline: inputs: - name: dummy dummy: '{"endpoint":"localhost", "value":"something"}' tag: dummy filters: - name: grep match: '*' logical_op: or regex: - value something - value error outputs: - name: stdout ``` {% endtab %} {% endtabs %} The output looks similar to: ``` Fluent Bit v2.0.9 * Copyright (C) 2015-2022 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io [2023/01/22 09:46:49] [ info] [fluent bit] version=2.0.9, commit=16eae10786, pid=33268 [2023/01/22 09:46:49] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128 [2023/01/22 09:46:49] [ info] [cmetrics] version=0.5.8 [2023/01/22 09:46:49] [ info] [ctraces ] version=0.2.7 [2023/01/22 09:46:49] [ info] [input:dummy:dummy.0] initializing [2023/01/22 09:46:49] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only) [2023/01/22 09:46:49] [ info] [filter:grep:grep.0] OR mode [2023/01/22 09:46:49] [ info] [sp] stream processor started [2023/01/22 09:46:49] [ info] [output:stdout:stdout.0] worker #0 started [0] dummy: [1674348410.558341857, {"endpoint"=>"localhost", "value"=>"something"}] [0] dummy: [1674348411.546425499, {"endpoint"=>"localhost", "value"=>"something"}] ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.fluentbit.io/manual/3.2/pipeline/filters/grep.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.