> For the complete documentation index, see [llms.txt](https://docs.fluentbit.io/manual/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.fluentbit.io/manual/data-pipeline/parsers/regular-expression.md).

# Regular expression format

Use the *regular expression* parser format to create custom parsers with Ruby regular expressions. These regular expressions use named capture to define which content belongs to which key name.

Use [Tail multiline](/manual/data-pipeline/inputs/tail.md#multiline-support) when you need to support regular expressions across multiple lines from a `tail`. The Tail input plugin treats each line as a separate entity.

{% hint style="warning" %}
This parser uses Onigmo, which is a backtracking regular expression's engine. When using complex regular expression patterns, Onigmo can take a long time to perform pattern matching. This can cause a [regular expression denial of service (ReDoS)](https://community.owasp.org/attacks/Regular_expression_Denial_of_Service_-_ReDoS).
{% endhint %}

Setting the format to regular expressions requires a `regex` configuration key.

For available configuration parameters, see [Configuring custom parsers](/manual/data-pipeline/parsers/configuring-parser.md).

## Regular expression work limit

In v5.2 or later, Fluent Bit limits how much work the regular expression engine can do for each search, which the engine calls the match operation limit. The limit applies to every regular expression in Fluent Bit, not only to parsers. A search can use up to 10,000,000 engine steps. For a search that scans the text, the steps from all starting positions count toward the same limit. The limit is set when Fluent Bit is built and can't be configured.

When a search reaches the limit, it fails, and Fluent Bit logs an error such as `[regex] search failed: -17`, or `[router] regex match failed: -17` for tag matching. The code `-17` means the limit was reached. Most features treat the failed search as no match, so the effect depends on where the expression is used:

* The [Parser](/manual/data-pipeline/filters/parser.md) filter keeps the original record and tries the next parser, if any.
* The [Syslog](/manual/data-pipeline/inputs/syslog.md) input logs the warning `error parsing log message with parser '<name>'` and drops the message.
* A [Grep](/manual/data-pipeline/filters/grep.md) filter `regex` rule drops the record, and an `exclude` rule keeps it.
* A [conditional routing](/manual/data-pipeline/router.md#conditional-routing) `regex` rule evaluates to false, and a `not_regex` rule evaluates to true.
* A `match_regex` pattern on a filter or output doesn't match the tag, so the filter isn't applied, or the records aren't routed to the output. Filters match the tag each time records are added, and outputs each time a chunk is created, so the error can repeat many times for the same tag.
* A [multiline](/manual/data-pipeline/parsers/multiline-parsing.md) rule treats the line as not matching.
* A [Tail](/manual/data-pipeline/inputs/tail.md) input `tag_regex` that fails for a filename prevents that file from being read.
* Some features don't treat a failed search as no match. A failed search can count as a match in [Sampling](/manual/data-pipeline/processors/sampling.md) processor attribute conditions. The same applies to the include, exclude, ignore, allow, and deny lists of the [Node exporter metrics](/manual/data-pipeline/inputs/node-exporter-metrics.md), [Process exporter metrics](/manual/data-pipeline/inputs/process-exporter-metrics.md), and [Windows exporter metrics](/manual/data-pipeline/inputs/windows-exporter-metrics.md) inputs. It also applies to the name and type filters of the [Thermal](/manual/data-pipeline/inputs/thermal.md) input. Depending on the list, the item can then be included or excluded unexpectedly. If you use regular expressions in these features, check the logs for `search failed` errors.

{% hint style="warning" %}
A regular expression can run on data from sources you don't control, such as record fields or tags received by the [Forward](/manual/data-pipeline/inputs/forward.md) input. Crafted values can then reach the limit on purpose and change the outcome. For example, they can skip a filter, or keep a record that a Grep `exclude` rule should drop, because a failed `exclude` search keeps the record. For these expressions, use anchored patterns, and avoid nested quantifiers such as `(a+)+` and repeated alternatives that can match the same text, such as `(a|a)*` or `(\w|\d)+`. These alternatives can reach the limit even in an anchored pattern without nested quantifiers. Don't rely on regular expression matching alone for filtering that must always apply.
{% endhint %}

## Configuration parameters

The `regex` parser supports the following format-specific configuration parameter:

| Key                 | Description                                                | Default |
| ------------------- | ---------------------------------------------------------- | ------- |
| `skip_empty_values` | If enabled, the parser ignores empty values of the record. | `true`  |

Fluent Bit uses the [Onigmo](https://github.com/k-takata/Onigmo) regular expression library in Ruby mode.

You can use only alphanumeric characters and underscore in group names. For example, a group name like `(?<user-name>.*)` causes an error due to the invalid dash (`-`) character. Use the [Rubular](https://rubular.com/) web editor to test your expressions.

The following parser configuration example provides rules that can be applied to an Apache HTTP Server log entry:

{% tabs %}
{% tab title="parsers.yaml" %}

```yaml
parsers:
  - name: apache
    format: regex
    regex: '^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$'
    time_key: time
    time_format: '%d/%b/%Y:%H:%M:%S %z'
    types: pid:integer size:integer
```

{% endtab %}

{% tab title="parsers.conf" %}

```
[PARSER]
  Name   apache
  Format regex
  Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
  Time_Key time
  Time_Format %d/%b/%Y:%H:%M:%S %z
  Types code:integer size:integer
```

{% endtab %}
{% endtabs %}

As an example, review the following Apache HTTP Server log entry:

```
192.168.2.20 - - [29/Jul/2015:10:27:10 -0300] "GET /cgi-bin/try/ HTTP/1.0" 200 3395
```

This log entry doesn't provide a defined structure for Fluent Bit. Enabling the proper parser can help to make a structured representation of the entry:

```
[1154104030, {"host"=>"192.168.2.20",
              "user"=>"-",
              "method"=>"GET",
              "path"=>"/cgi-bin/try/",
              "code"=>"200",
              "size"=>"3395",
              "referer"=>"",
              "agent"=>""
              }
]
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.fluentbit.io/manual/data-pipeline/parsers/regular-expression.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
