Fluent Bit: Official Manual
SlackGitHubCommunity MeetingsSandbox and LabsWebinars
1.0
1.0
  • Introduction
  • About
    • Why ?
    • Fluentd & Fluent Bit
    • License
  • Installation
    • Supported Platforms
    • Requirements
    • Download Sources
    • Build and Install
    • Build with Static Configuration
    • Docker Images
    • Kubernetes
    • TD Agent Bit
    • Debian Packages
    • Ubuntu Packages
    • CentOS Packages
    • Raspberry Pi
    • Yocto Project
    • Unit Tests
  • Getting Started
    • Service
    • Input
    • Parser
    • Filter
    • Buffer
    • Routing
    • Output
  • Configuration
    • Configuration Schema
    • Configuration File
    • Configuration Variables
    • Configuration Commands
    • Buffering / Storage
    • Monitoring
    • Unit Sizes
    • TLS / SSL
    • Backpressure
    • Memory Usage
    • Upstream Servers
    • Scheduler
  • Service
  • Input Plugins
    • CPU Usage
    • Disk Usage
    • Dummy
    • Exec
    • Forward
    • Head
    • Health
    • Kernel Log Buffer
    • Memory Usage
    • MQTT
    • Network Traffic
    • Process
    • Random
    • Serial Interface
    • Standard Input
    • Syslog
    • Systemd
    • Tail
    • TCP
  • Parsers
    • JSON Parser
    • Regular Expression Parser
    • Decoders
  • Filter Plugins
    • Grep
    • Kubernetes
    • Lua
    • Parser
    • Record Modifier
    • Standard Output
    • Throttle
    • Nest
    • Modify
  • Output Plugins
    • Azure
    • BigQuery
    • Counter
    • Elasticsearch
    • File
    • FlowCounter
    • Forward
    • HTTP
    • InfluxDB
    • Kafka
    • Kafka REST Proxy
    • NATS
    • Null
    • Stackdriver
    • Standard Output
    • Splunk
    • Treasure Data
  • Fluent Bit for Developers
    • Library API
    • Ingest Records Manually
Powered by GitBook
On this page

Was this helpful?

Export as PDF
  1. Parsers

Regular Expression Parser

Last updated 4 years ago

Was this helpful?

The regex parser allows us to define a custom Ruby Regular Expression that will use a named capture feature to define which content belongs to which key name.

Fluent Bit uses regular expression library on Ruby mode, for testing purposes you can use the following web editor to test your expressions:

Important: do not attempt to add multiline support in your regular expressions if you are using input plugin since each line is handled as a separated entity. Instead use Tail support configuration feature.

Note: understanding how regular expressions works is out of the scope of this content.

From a configuration perspective, when the format is set to regex, is mandatory and expected that a Regex configuration key exists.

The following parser configuration example aims to provide rules that can be applied to an Apache HTTP Server log entry:

[PARSER]
    Name   apache
    Format regex
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

As an example, takes the following Apache HTTP Server log entry:

192.168.2.20 - - [29/Jul/2015:10:27:10 -0300] "GET /cgi-bin/try/ HTTP/1.0" 200 3395

The above content do not provide a defined structure for Fluent Bit, but enabling the proper parser we can help to make a structured representation of it:

[1154104030, {"host"=>"192.168.2.20",
              "user"=>"-",
              "method"=>"GET",
              "path"=>"/cgi-bin/try/",
              "code"=>"200",
              "size"=>"3395",
              "referer"=>"",
              "agent"=>""
              }
]

In order to understand, learn and test regular expressions like the example above, we suggest you try the following Ruby Regular Expression Editor:

http://rubular.com/r/X7BH0M4Ivm
Onigmo
http://rubular.com/
Tail
Multiline