Regular Expression

The regex parser allows to define a custom Ruby Regular Expression that will use a named capture feature to define which content belongs to which key name.
Fluent Bit uses Onigmo regular expression library on Ruby mode, for testing purposes you can use the following web editor to test your expressions:
​http://rubular.com/​
Important: do not attempt to add multiline support in your regular expressions if you are using Tail input plugin since each line is handled as a separated entity. Instead use Tail Multiline support configuration feature.
Security Warning: Onigmo is a backtracking regex engine. You need to be careful not to use expensive regex patterns, or Onigmo can take very long time to perform pattern matching. For details, please read the article "ReDoS" on OWASP.
Note: understanding how regular expressions works is out of the scope of this content.
From a configuration perspective, when the format is set to regex, is mandatory and expected that a Regex configuration key exists.
Configuration Parameters
The regex parser supports the following configuration parameters.
Key
Description
Default Value
Skip_Empty_Values
If enabled, the parser ignores empty value of the record.
True
The following parser configuration example aims to provide rules that can be applied to an Apache HTTP Server log entry:
1
[PARSER]
2
    Name   apache
3
    Format regex
4
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
5
    Time_Key time
6
    Time_Format %d/%b/%Y:%H:%M:%S %z
7
    Types code:integer size:integer
Copied!
As an example, takes the following Apache HTTP Server log entry:
1
192.168.2.20 - - [29/Jul/2015:10:27:10 -0300] "GET /cgi-bin/try/ HTTP/1.0" 200 3395
Copied!
The above content do not provide a defined structure for Fluent Bit, but enabling the proper parser we can help to make a structured representation of it:
1
[1154104030, {"host"=>"192.168.2.20",
2
              "user"=>"-",
3
              "method"=>"GET",
4
              "path"=>"/cgi-bin/try/",
5
              "code"=>"200",
6
              "size"=>"3395",
7
              "referer"=>"",
8
              "agent"=>""
9
              }
10
]
Copied!
A common pitfall is that you cannot use characters other than alphabets, numbers and underscore in group names. For example, a group name like (?<user-name>.*) will cause an error due to containing an invalid character (-).
In order to understand, learn and test regular expressions like the example above, we suggest you try the following Ruby Regular Expression Editor: http://rubular.com/r/X7BH0M4Ivm​

JSON

LTSV

Last modified 4mo ago

Export as PDF

Copy link

Contents

Configuration Parameters